[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:51:58 dist-test-slave-v5vl krb5kdc[17950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:51:58 dist-test-slave-v5vl krb5kdc[17950](info): set up 2 sockets
Sep 25 14:51:58 dist-test-slave-v5vl krb5kdc[17950](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 14:51:59.353526 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:51:59 dist-test-slave-v5vl krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:51:59.524073 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:40979
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:40979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:59.842450 17963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:59.842792 17963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:59.842955 17963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:59.850132 17963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:59.850281 17963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:59.850397 17963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:59.850497 17963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:59.850584 17963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:59.860515 17963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:40979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:40979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:51:59.861812 17963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:59.863135 17963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:59.874974 17968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:59.875018 17969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:59.875041 17971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:59.875856 17963 server_base.cc:1034] running on GCE node
I20260925 14:51:59.876502 17963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:59.877737 17963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:59.878964 17963 hybrid_clock.cc:648] HybridClock initialized: now 1790347919878914 us; error 42 us; skew 500 ppm
I20260925 14:51:59.879256 17963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:51:59 dist-test-slave-v5vl krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347919, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:59.882153 17963 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:51:59.883510 17963 webserver.cc:460] Webserver started at http://127.17.133.254:43639/ using document root <none> and password file <none>
I20260925 14:51:59.884163 17963 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:59.884294 17963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:59.884631 17963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:59.887427 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "409a0276f6604e4c9b3b1885971f6d27"
format_stamp: "Formatted at 2026-09-25 14:51:59 on dist-test-slave-v5vl"
server_key: "cbe7a9589af050392f4a1db58c8eacc0"
server_key_iv: "47bc4bcfcffc2ac362fd43671d4d6b3c"
server_key_version: "encryptionkey@0"
I20260925 14:51:59.888186 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "409a0276f6604e4c9b3b1885971f6d27"
format_stamp: "Formatted at 2026-09-25 14:51:59 on dist-test-slave-v5vl"
server_key: "cbe7a9589af050392f4a1db58c8eacc0"
server_key_iv: "47bc4bcfcffc2ac362fd43671d4d6b3c"
server_key_version: "encryptionkey@0"
I20260925 14:51:59.893589 17963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 14:51:59.896812 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:59.898058 17963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:51:59.898231 17963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "409a0276f6604e4c9b3b1885971f6d27"
format_stamp: "Formatted at 2026-09-25 14:51:59 on dist-test-slave-v5vl"
server_key: "cbe7a9589af050392f4a1db58c8eacc0"
server_key_iv: "47bc4bcfcffc2ac362fd43671d4d6b3c"
server_key_version: "encryptionkey@0"
I20260925 14:51:59.898418 17963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:59.921152 17963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:59.924953 17963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:59.925210 17963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:59.943602 17963 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:40979
I20260925 14:51:59.943629 18029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:40979 every 8 connection(s)
I20260925 14:51:59.944929 17963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:59.948326 18030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:59.950721 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 17963
I20260925 14:51:59.951012 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:51:59.952836 17943 external_mini_cluster.cc:1428] Setting key e1cd8372b0da7a130560379fa6a486ea
I20260925 14:51:59.959326 18030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27: Bootstrap starting.
I20260925 14:51:59.962051 18030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:59.963191 18030 log.cc:824] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:59.965817 18030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27: No bootstrap required, opened a new log
I20260925 14:51:59.973373 18030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409a0276f6604e4c9b3b1885971f6d27" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40979 } }
I20260925 14:51:59.973762 18030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:59.973870 18030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409a0276f6604e4c9b3b1885971f6d27, State: Initialized, Role: FOLLOWER
I20260925 14:51:59.974511 18030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409a0276f6604e4c9b3b1885971f6d27" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40979 } }
I20260925 14:51:59.974772 18030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:59.974932 18030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:59.975134 18030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:59.977334 18030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409a0276f6604e4c9b3b1885971f6d27" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40979 } }
I20260925 14:51:59.977826 18030 leader_election.cc:302] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 409a0276f6604e4c9b3b1885971f6d27; no voters:
I20260925 14:51:59.978448 18030 leader_election.cc:288] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:59.978566 18035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [term 1 FOLLOWER]: Leader election won for term 1
Sep 25 14:51:59 dist-test-slave-v5vl krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:51:59.979846 18035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [term 1 LEADER]: Becoming Leader. State: Replica: 409a0276f6604e4c9b3b1885971f6d27, State: Running, Role: LEADER
I20260925 14:51:59.980444 18035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409a0276f6604e4c9b3b1885971f6d27" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40979 } }
I20260925 14:51:59.980865 18030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:59.984334 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 409a0276f6604e4c9b3b1885971f6d27. Latest consensus state: current_term: 1 leader_uuid: "409a0276f6604e4c9b3b1885971f6d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409a0276f6604e4c9b3b1885971f6d27" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40979 } } }
I20260925 14:51:59.984764 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:59.985222 18036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "409a0276f6604e4c9b3b1885971f6d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409a0276f6604e4c9b3b1885971f6d27" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40979 } } }
I20260925 14:51:59.985509 18036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:59.996300 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:59.956285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.17.133.254:40979)
0925 14:51:59.957165 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:59.957185 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:59.958113 (+ 928us) server_negotiation.cc:407] Connection header received
0925 14:51:59.966867 (+ 8754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:59.966885 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:59.967215 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:59.967612 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:59.970332 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:59.972195 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:59.973922 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:59.974645 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:59.974853 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:59.981348 (+ 6495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:59.981360 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:59.981380 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:59.981432 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:59.986589 (+ 5157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:59.987966 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:59.987978 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:59.987990 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:59.988283 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:59.988971 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:59.988979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:59.988984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:59.989884 (+ 900us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:59.990249 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:59.993909 (+ 3660us) server_negotiation.cc:299] Negotiation successful
0925 14:51:59.994456 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":419,"threads_started":1}
I20260925 14:51:59.999730 18043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:00.005119 18043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:00.014833 18043 catalog_manager.cc:1347] Generated new cluster ID: 2b2eab474bfb4985b9d5aba29f2754c2
I20260925 14:52:00.014951 18043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:00.046262 18043 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:00.048161 18043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:00.061700 18043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 409a0276f6604e4c9b3b1885971f6d27: Generated new TSK 0
I20260925 14:52:00.062753 18043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:52:00.112346 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40979
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:00.451588 18056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:00.451867 18056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:00.452031 18056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:00.459098 18056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:00.459224 18056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:00.459419 18056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:00.468487 18056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40979
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:00.469693 18056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:00.470942 18056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:00.483887 18064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:00.483963 18061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:00.483958 18062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:00.484978 18056 server_base.cc:1034] running on GCE node
I20260925 14:52:00.485463 18056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:00.486270 18056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:00.487555 18056 hybrid_clock.cc:648] HybridClock initialized: now 1790347920487483 us; error 50 us; skew 500 ppm
I20260925 14:52:00.487874 18056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:00 dist-test-slave-v5vl krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347920, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:00.491345 18056 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:52:00.492883 18056 webserver.cc:460] Webserver started at http://127.17.133.193:41369/ using document root <none> and password file <none>
I20260925 14:52:00.493600 18056 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:00.493743 18056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:00.494057 18056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:00.496899 18056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "200d438153c2423aa4cc4623bffe5f73"
format_stamp: "Formatted at 2026-09-25 14:52:00 on dist-test-slave-v5vl"
server_key: "bfae4d97bc520a6c3c793fda5c10aaa8"
server_key_iv: "38049ae08e8dbaabe6b7fb5a13c0f6ec"
server_key_version: "encryptionkey@0"
I20260925 14:52:00.497676 18056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "200d438153c2423aa4cc4623bffe5f73"
format_stamp: "Formatted at 2026-09-25 14:52:00 on dist-test-slave-v5vl"
server_key: "bfae4d97bc520a6c3c793fda5c10aaa8"
server_key_iv: "38049ae08e8dbaabe6b7fb5a13c0f6ec"
server_key_version: "encryptionkey@0"
I20260925 14:52:00.502733 18056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:52:00.505995 18071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:00.507539 18056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:52:00.507706 18056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "200d438153c2423aa4cc4623bffe5f73"
format_stamp: "Formatted at 2026-09-25 14:52:00 on dist-test-slave-v5vl"
server_key: "bfae4d97bc520a6c3c793fda5c10aaa8"
server_key_iv: "38049ae08e8dbaabe6b7fb5a13c0f6ec"
server_key_version: "encryptionkey@0"
I20260925 14:52:00.507928 18056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:00.550354 18056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:00.555028 18056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:00.555327 18056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:00.556303 18056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:00.557628 18056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:00.557696 18056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:00.557834 18056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:00.557907 18056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:00.591612 18056 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:34591
I20260925 14:52:00.591655 18183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:34591 every 8 connection(s)
I20260925 14:52:00.592995 18056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:00.600250 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 18056
I20260925 14:52:00.600477 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:00.601109 17943 external_mini_cluster.cc:1428] Setting key 958467bd96782046165315f0763a8082
Sep 25 14:52:00 dist-test-slave-v5vl krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347920, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:00.613898 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:00.595526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:44079 (local address 127.17.133.254:40979)
0925 14:52:00.595936 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:00.595940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:00.596986 (+ 1046us) server_negotiation.cc:407] Connection header received
0925 14:52:00.599260 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:00.599265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:00.599346 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:00.599475 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:00.601067 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:00.601997 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:00.603676 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:00.604090 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:00.604224 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:00.607706 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:00.607713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:00.607717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:00.607759 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:00.610460 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:00.611516 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:00.611519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:00.611522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:00.611631 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:00.612083 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:00.612087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:00.612091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:00.612338 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:00.612442 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:00.613434 (+ 992us) server_negotiation.cc:299] Negotiation successful
0925 14:52:00.613642 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":216,"threads_started":1}
I20260925 14:52:00.615553 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:00.596114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40979 (local address 127.17.133.193:44079)
0925 14:52:00.596817 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:00.596860 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:00.598991 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:00.599729 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:00.599737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:00.600221 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:00.600901 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:00.600918 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:00.602181 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:00.602189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:00.603525 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:00.603536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:00.604323 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:00.604333 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:00.604567 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:00.605337 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:00.605358 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:00.607466 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:00.610611 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:00.610620 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:00.610638 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:00.611373 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:00.611738 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:00.611743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:00.611747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:00.611935 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:00.612458 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:00.612468 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:00.612996 (+ 528us) client_negotiation.cc:769] Sending connection context
0925 14:52:00.613341 (+ 345us) client_negotiation.cc:240] Negotiation successful
0925 14:52:00.614046 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:52:00.617180 18184 heartbeater.cc:347] Connected to a master server at 127.17.133.254:40979
I20260925 14:52:00.617447 18184 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:00.618165 18184 heartbeater.cc:511] Master 127.17.133.254:40979 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
I20260925 14:52:00.620157 17995 ts_manager.cc:194] Registered new tserver with Master: 200d438153c2423aa4cc4623bffe5f73 (127.17.133.193:34591)
I20260925 14:52:00.621676 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:44079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:52:00.640098 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40979
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:00.958933 18190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:00.959270 18190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:00.959447 18190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:00.966715 18190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:00.966863 18190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:00.967058 18190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:00.977201 18190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40979
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:00.978509 18190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:00.979871 18190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:00.993225 18196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:00.993182 18195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:00.993260 18198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:00.994001 18190 server_base.cc:1034] running on GCE node
I20260925 14:52:00.994581 18190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:00.995458 18190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:00.996683 18190 hybrid_clock.cc:648] HybridClock initialized: now 1790347920996638 us; error 48 us; skew 500 ppm
I20260925 14:52:00.996984 18190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:00 dist-test-slave-v5vl krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347920, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:01.000043 18190 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:52:01.001710 18190 webserver.cc:460] Webserver started at http://127.17.133.194:39907/ using document root <none> and password file <none>
I20260925 14:52:01.002419 18190 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:01.002576 18190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:01.002894 18190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:01.005961 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "cde86054aabf49d4a61d98e20bc08216"
format_stamp: "Formatted at 2026-09-25 14:52:01 on dist-test-slave-v5vl"
server_key: "bc033aa521b2b70100f671a712ee57a3"
server_key_iv: "e1a30eb0a2cd644c6738aa4ba64fca48"
server_key_version: "encryptionkey@0"
I20260925 14:52:01.006685 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "cde86054aabf49d4a61d98e20bc08216"
format_stamp: "Formatted at 2026-09-25 14:52:01 on dist-test-slave-v5vl"
server_key: "bc033aa521b2b70100f671a712ee57a3"
server_key_iv: "e1a30eb0a2cd644c6738aa4ba64fca48"
server_key_version: "encryptionkey@0"
I20260925 14:52:01.011781 18190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 14:52:01.015537 18205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:01.016834 18190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:01.016995 18190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "cde86054aabf49d4a61d98e20bc08216"
format_stamp: "Formatted at 2026-09-25 14:52:01 on dist-test-slave-v5vl"
server_key: "bc033aa521b2b70100f671a712ee57a3"
server_key_iv: "e1a30eb0a2cd644c6738aa4ba64fca48"
server_key_version: "encryptionkey@0"
I20260925 14:52:01.017177 18190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:01.035526 18190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:01.039134 18190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:01.039451 18190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:01.040380 18190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:01.041640 18190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:01.041738 18190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:01.041880 18190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:01.041949 18190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:01.076632 18190 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:39015
I20260925 14:52:01.076699 18317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:39015 every 8 connection(s)
I20260925 14:52:01.077953 18190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:01.085525 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 18190
I20260925 14:52:01.085754 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:01.086180 17943 external_mini_cluster.cc:1428] Setting key 9629108f0b989d2b2adc5b8d38c47d89
Sep 25 14:52:01 dist-test-slave-v5vl krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347920, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:01.100442 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:01.080518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:47737 (local address 127.17.133.254:40979)
0925 14:52:01.080701 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:01.080706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:01.081947 (+ 1241us) server_negotiation.cc:407] Connection header received
0925 14:52:01.083757 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:01.083761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:01.083826 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:01.083913 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:01.085601 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.086433 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.088311 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.088767 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.088860 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:01.093446 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:01.093455 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:01.093460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:01.093505 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:01.097204 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.097901 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.097908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.097914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.098065 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.098470 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.098477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.098483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.098920 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:01.099067 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:01.099705 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 14:52:01.099998 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:52:01.101186 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:01.081085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40979 (local address 127.17.133.194:47737)
0925 14:52:01.081790 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:01.081826 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:01.083511 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:01.084115 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:01.084123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:01.084690 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:01.085406 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:01.085431 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.086580 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.086586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:01.088181 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:01.088196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.088991 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.089002 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:01.089329 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:01.090520 (+ 1191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:01.090560 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:01.093214 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:01.097331 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:01.097337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:01.097350 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:01.097758 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:01.098165 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:01.098168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:01.098170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:01.098355 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:01.099053 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:01.099058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:01.099348 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 14:52:01.099558 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 14:52:01.100016 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":273,"threads_started":1}
I20260925 14:52:01.102746 18318 heartbeater.cc:347] Connected to a master server at 127.17.133.254:40979
I20260925 14:52:01.103134 18318 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:01.104053 18318 heartbeater.cc:511] Master 127.17.133.254:40979 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
I20260925 14:52:01.105624 17995 ts_manager.cc:194] Registered new tserver with Master: cde86054aabf49d4a61d98e20bc08216 (127.17.133.194:39015)
I20260925 14:52:01.106524 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:47737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:52:01.127058 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40979
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:01.449920 18323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:01.450176 18323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:01.450282 18323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:01.457319 18323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:01.457463 18323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:01.457617 18323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:01.466495 18323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40979
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:01.467764 18323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:01.469059 18323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:01.482684 18331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:01.482760 18329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:01.482759 18328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:01.483551 18323 server_base.cc:1034] running on GCE node
I20260925 14:52:01.484138 18323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:01.485054 18323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:01.486266 18323 hybrid_clock.cc:648] HybridClock initialized: now 1790347921486219 us; error 59 us; skew 500 ppm
I20260925 14:52:01.486550 18323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:01 dist-test-slave-v5vl krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347921, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:01.489415 18323 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:52:01.490782 18323 webserver.cc:460] Webserver started at http://127.17.133.195:42915/ using document root <none> and password file <none>
I20260925 14:52:01.491425 18323 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:01.491578 18323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:01.491907 18323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:01.494634 18323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "a137d4c3080c4d708b199d307a7fb831"
format_stamp: "Formatted at 2026-09-25 14:52:01 on dist-test-slave-v5vl"
server_key: "656428094d585f77284b2081c61d2bb4"
server_key_iv: "470fca80fb93d055bd45dea63b1df3fd"
server_key_version: "encryptionkey@0"
I20260925 14:52:01.495764 18323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "a137d4c3080c4d708b199d307a7fb831"
format_stamp: "Formatted at 2026-09-25 14:52:01 on dist-test-slave-v5vl"
server_key: "656428094d585f77284b2081c61d2bb4"
server_key_iv: "470fca80fb93d055bd45dea63b1df3fd"
server_key_version: "encryptionkey@0"
I20260925 14:52:01.500768 18323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 14:52:01.504104 18338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:01.505362 18323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:01.505544 18323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "a137d4c3080c4d708b199d307a7fb831"
format_stamp: "Formatted at 2026-09-25 14:52:01 on dist-test-slave-v5vl"
server_key: "656428094d585f77284b2081c61d2bb4"
server_key_iv: "470fca80fb93d055bd45dea63b1df3fd"
server_key_version: "encryptionkey@0"
I20260925 14:52:01.505752 18323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:01.524555 18323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:01.528959 18323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:01.529214 18323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:01.530102 18323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:01.531332 18323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:01.531410 18323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:01.531494 18323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:01.531536 18323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:01.566748 18323 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:34515
I20260925 14:52:01.566787 18450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:34515 every 8 connection(s)
I20260925 14:52:01.568075 18323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:01.572774 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 18323
I20260925 14:52:01.572968 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:01.573482 17943 external_mini_cluster.cc:1428] Setting key 4f4e02236772755d02610aabec37019e
Sep 25 14:52:01 dist-test-slave-v5vl krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347921, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:01.588114 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:01.570562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:53653 (local address 127.17.133.254:40979)
0925 14:52:01.570734 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:01.570738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:01.571947 (+ 1209us) server_negotiation.cc:407] Connection header received
0925 14:52:01.574522 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:01.574529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:01.574632 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:01.574714 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:01.576752 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.577500 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.578966 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.579324 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.579429 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:01.583160 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:01.583165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:01.583168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:01.583199 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:01.585537 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.586142 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.586146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.586148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.586258 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.586585 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.586587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.586589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.586820 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:01.586922 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:01.587554 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 14:52:01.587737 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:52:01.589169 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:01.571103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40979 (local address 127.17.133.195:53653)
0925 14:52:01.571779 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:01.571820 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:01.574176 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:01.574992 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:01.575005 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:01.575820 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:01.576490 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:01.576508 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.577636 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.577640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:01.578852 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:01.578862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.579957 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.579963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:01.580157 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:01.580862 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:01.580887 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:01.582964 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:01.585651 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:01.585657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:01.585668 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:01.586052 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:01.586343 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:01.586346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:01.586349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:01.586499 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:01.586917 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:01.586922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:01.587210 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 14:52:01.587504 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 14:52:01.587992 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":259,"threads_started":1}
I20260925 14:52:01.590451 18451 heartbeater.cc:347] Connected to a master server at 127.17.133.254:40979
I20260925 14:52:01.590700 18451 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:01.591405 18451 heartbeater.cc:511] Master 127.17.133.254:40979 requested a full tablet report, sending...
I20260925 14:52:01.592778 17995 ts_manager.cc:194] Registered new tserver with Master: a137d4c3080c4d708b199d307a7fb831 (127.17.133.195:34515)
I20260925 14:52:01.593554 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:53653
I20260925 14:52:01.601353 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:52:01 dist-test-slave-v5vl krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347921, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 14:52:01.624536 18184 heartbeater.cc:503] Master 127.17.133.254:40979 was elected leader, sending a full tablet report...
Sep 25 14:52:01 dist-test-slave-v5vl krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347921, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:01.646355 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:01.631198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.17.133.193:34591)
0925 14:52:01.632254 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:01.632266 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:01.632318 (+ 52us) server_negotiation.cc:407] Connection header received
0925 14:52:01.632555 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:01.632562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:01.632827 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:01.632977 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:01.634307 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.635248 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.636402 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.637027 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.637184 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:01.641208 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:01.641220 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:01.641235 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:01.641272 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:01.643907 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.644524 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.644531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.644534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.644762 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.645123 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.645126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.645127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.645435 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:01.645654 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:01.645906 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 14:52:01.646088 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":336,"threads_started":1}
W20260925 14:52:01.647610 18099 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53224
Sep 25 14:52:01 dist-test-slave-v5vl krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:52:01 dist-test-slave-v5vl krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:01.688644 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:01.675496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.17.133.193:34591)
0925 14:52:01.675692 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:01.675700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:01.676108 (+ 408us) server_negotiation.cc:407] Connection header received
0925 14:52:01.676289 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:01.676293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:01.676366 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:01.676445 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:01.677683 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.678318 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.679387 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.679759 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.679841 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:01.683936 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:01.683940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:01.683943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:01.683972 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:01.686269 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.686855 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.686862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.686870 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.686978 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.687319 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.687326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.687329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.687579 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:01.687811 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:01.688091 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:52:01.688288 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:52:01.689981 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 18056
I20260925 14:52:01.701463 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:01.690808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.17.133.193:34591)
0925 14:52:01.691013 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:01.691018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:01.691237 (+ 219us) server_negotiation.cc:407] Connection header received
0925 14:52:01.691335 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:01.691338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:01.691386 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:01.691458 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:01.692480 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.693155 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.694242 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.694642 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.694724 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:01.697118 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:01.697122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:01.697125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:01.697166 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:01.699371 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.699933 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.699936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.699938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.700048 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.700329 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.700331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.700333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.700542 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:01.700765 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:01.700980 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 14:52:01.701112 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:52:01.702117 18179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:01.784561 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 18056
I20260925 14:52:01.800735 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 18190
Sep 25 14:52:01 dist-test-slave-v5vl krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:52:01.815671 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:01.801671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.17.133.194:39015)
0925 14:52:01.802179 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:01.802185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:01.802209 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:52:01.802271 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:01.802277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:01.802479 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:01.802632 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:01.803770 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.804887 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.805986 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.806493 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.806584 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:01.810567 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:01.810581 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:01.810604 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:01.810640 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:01.813328 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.813920 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.813926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.813929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.814042 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.814412 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.814414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.814416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.814719 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:01.814928 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:01.815211 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 14:52:01.815389 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":269,"threads_started":1}
I20260925 14:52:01.816944 18313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:01.896867 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 18190
I20260925 14:52:01.918310 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 18323
Sep 25 14:52:01 dist-test-slave-v5vl krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:52:01.933614 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:01.919282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.17.133.195:34515)
0925 14:52:01.919939 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:01.919946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:01.919971 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:52:01.920030 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:01.920037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:01.920284 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:01.920430 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:01.921850 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.922848 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.924014 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:01.924541 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:01.924742 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:01.928523 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:01.928533 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:01.928547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:01.928631 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:01.931487 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.932003 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.932009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.932011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.932134 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:01.932406 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:01.932409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:01.932411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:01.932759 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:01.932973 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:01.933177 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 14:52:01.933348 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":355,"threads_started":1}
I20260925 14:52:01.934844 18446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:02.017108 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 18323
I20260925 14:52:02.033215 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 17963
I20260925 14:52:02.034308 18025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:02.132639 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 17963
2026-09-25T14:52:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3871 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:02 dist-test-slave-v5vl krb5kdc[18472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:02 dist-test-slave-v5vl krb5kdc[18472](info): set up 2 sockets
Sep 25 14:52:02 dist-test-slave-v5vl krb5kdc[18472](info): commencing operation
krb5kdc: starting...
W20260925 14:52:03.193118 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:03 dist-test-slave-v5vl krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:52:03.361361 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:35507
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:35507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:03.669929 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:03.670192 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:03.670334 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:03.677317 18485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:03.677408 18485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:03.677506 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:03.677600 18485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:03.677678 18485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:03.687211 18485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:35507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:35507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:03.688385 18485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:03.689596 18485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:03.700760 18490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:03.700706 18493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:03.700796 18491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:03.701771 18485 server_base.cc:1034] running on GCE node
I20260925 14:52:03.702447 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:03.703653 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:03.704873 18485 hybrid_clock.cc:648] HybridClock initialized: now 1790347923704820 us; error 43 us; skew 500 ppm
I20260925 14:52:03.705179 18485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:03 dist-test-slave-v5vl krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:03.710800 18485 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:52:03.712208 18485 webserver.cc:460] Webserver started at http://127.17.133.254:43397/ using document root <none> and password file <none>
I20260925 14:52:03.712932 18485 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:03.713066 18485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:03.713366 18485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:03.716136 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "9539781590fd41d2b41376aeea1343e6"
format_stamp: "Formatted at 2026-09-25 14:52:03 on dist-test-slave-v5vl"
server_key: "cf8ae5eebde8f45bf30e017876da631b"
server_key_iv: "de88e4a602952965b289e19048f36ebc"
server_key_version: "encryptionkey@0"
I20260925 14:52:03.716885 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "9539781590fd41d2b41376aeea1343e6"
format_stamp: "Formatted at 2026-09-25 14:52:03 on dist-test-slave-v5vl"
server_key: "cf8ae5eebde8f45bf30e017876da631b"
server_key_iv: "de88e4a602952965b289e19048f36ebc"
server_key_version: "encryptionkey@0"
I20260925 14:52:03.721417 18485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 14:52:03.724506 18500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:03.725802 18485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:52:03.725977 18485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "9539781590fd41d2b41376aeea1343e6"
format_stamp: "Formatted at 2026-09-25 14:52:03 on dist-test-slave-v5vl"
server_key: "cf8ae5eebde8f45bf30e017876da631b"
server_key_iv: "de88e4a602952965b289e19048f36ebc"
server_key_version: "encryptionkey@0"
I20260925 14:52:03.726154 18485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:03.746765 18485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:03.750607 18485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:03.750903 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:03.769472 18485 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:35507
I20260925 14:52:03.769491 18551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:35507 every 8 connection(s)
I20260925 14:52:03.770869 18485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:03.774559 18552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:03.776978 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 18485
I20260925 14:52:03.777192 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:03.777951 17943 external_mini_cluster.cc:1428] Setting key e5a0cfc497c2de71d9242b525cf04931
I20260925 14:52:03.786810 18552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6: Bootstrap starting.
I20260925 14:52:03.790575 18552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:03.791762 18552 log.cc:824] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6: Log is configured to *not* fsync() on all Append() calls
Sep 25 14:52:03 dist-test-slave-v5vl krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:03.795100 18552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6: No bootstrap required, opened a new log
I20260925 14:52:03.801352 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:03.779974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.17.133.254:35507)
0925 14:52:03.780799 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:03.780809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:03.780848 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:52:03.782397 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:03.782417 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:03.782779 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:03.783091 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:03.784816 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:03.786062 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:03.787718 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:03.788366 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:03.788486 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:03.793306 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:03.793315 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:03.793343 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:03.793388 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:03.797186 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:03.797800 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:03.797810 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:03.797823 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:03.798032 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:03.798370 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:03.798380 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:03.798396 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:03.799009 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:03.799291 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:03.799648 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 14:52:03.800019 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":406,"threads_started":1}
I20260925 14:52:03.804502 18552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539781590fd41d2b41376aeea1343e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 35507 } }
I20260925 14:52:03.804916 18552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:03.804986 18552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9539781590fd41d2b41376aeea1343e6, State: Initialized, Role: FOLLOWER
I20260925 14:52:03.805424 18552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539781590fd41d2b41376aeea1343e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 35507 } }
I20260925 14:52:03.805610 18552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:03.805696 18552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:03.805811 18552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:03.807782 18552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539781590fd41d2b41376aeea1343e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 35507 } }
I20260925 14:52:03.808183 18552 leader_election.cc:302] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9539781590fd41d2b41376aeea1343e6; no voters:
I20260925 14:52:03.808739 18552 leader_election.cc:288] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:03.808823 18557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:03.810011 18557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9539781590fd41d2b41376aeea1343e6, State: Running, Role: LEADER
I20260925 14:52:03.810559 18557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539781590fd41d2b41376aeea1343e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 35507 } }
I20260925 14:52:03.810875 18552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:03.813346 18559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9539781590fd41d2b41376aeea1343e6. Latest consensus state: current_term: 1 leader_uuid: "9539781590fd41d2b41376aeea1343e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539781590fd41d2b41376aeea1343e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 35507 } } }
I20260925 14:52:03.813462 18558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9539781590fd41d2b41376aeea1343e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539781590fd41d2b41376aeea1343e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 35507 } } }
I20260925 14:52:03.813719 18559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:03.813743 18558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:03.817263 18565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:03.821659 18565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:03.830265 18565 catalog_manager.cc:1347] Generated new cluster ID: e85e67b4ffe847469a38d1dd24fa8216
I20260925 14:52:03.830413 18565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:03.838189 18565 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:03.839864 18565 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:03.850852 18565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9539781590fd41d2b41376aeea1343e6: Generated new TSK 0
I20260925 14:52:03.851588 18565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:52:03.900265 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:35507
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:04.231045 18578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:04.231302 18578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:04.231396 18578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:04.238366 18578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:04.238487 18578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:04.238586 18578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:04.247439 18578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:35507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:04.248569 18578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:04.249850 18578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:04.262800 18583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:04.262887 18586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:04.262918 18584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:04.263734 18578 server_base.cc:1034] running on GCE node
I20260925 14:52:04.264231 18578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:04.265158 18578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:04.266364 18578 hybrid_clock.cc:648] HybridClock initialized: now 1790347924266311 us; error 53 us; skew 500 ppm
I20260925 14:52:04.266669 18578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:04 dist-test-slave-v5vl krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:04.269814 18578 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:52:04.271203 18578 webserver.cc:460] Webserver started at http://127.17.133.193:41589/ using document root <none> and password file <none>
I20260925 14:52:04.271818 18578 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:04.271914 18578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:04.272162 18578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:04.275695 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "3caba136d0de4d239080dd750c809c3d"
format_stamp: "Formatted at 2026-09-25 14:52:04 on dist-test-slave-v5vl"
server_key: "ed9f3965f092912baa4b6a160e23fe9d"
server_key_iv: "5fad6975d61a0a18d08d8716fd147613"
server_key_version: "encryptionkey@0"
I20260925 14:52:04.276700 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "3caba136d0de4d239080dd750c809c3d"
format_stamp: "Formatted at 2026-09-25 14:52:04 on dist-test-slave-v5vl"
server_key: "ed9f3965f092912baa4b6a160e23fe9d"
server_key_iv: "5fad6975d61a0a18d08d8716fd147613"
server_key_version: "encryptionkey@0"
I20260925 14:52:04.281978 18578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 14:52:04.285168 18593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:04.286427 18578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:04.286595 18578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "3caba136d0de4d239080dd750c809c3d"
format_stamp: "Formatted at 2026-09-25 14:52:04 on dist-test-slave-v5vl"
server_key: "ed9f3965f092912baa4b6a160e23fe9d"
server_key_iv: "5fad6975d61a0a18d08d8716fd147613"
server_key_version: "encryptionkey@0"
I20260925 14:52:04.286784 18578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:04.304306 18578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:04.308480 18578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:04.308802 18578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:04.309685 18578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:04.311070 18578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:04.311165 18578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:04.311327 18578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:04.311399 18578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:04.342576 18578 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:34443
I20260925 14:52:04.342617 18705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:34443 every 8 connection(s)
I20260925 14:52:04.343961 18578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:04.347640 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 18578
I20260925 14:52:04.347851 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:04.348646 17943 external_mini_cluster.cc:1428] Setting key c7b5134fdab8bb018061403c2409d4b7
Sep 25 14:52:04 dist-test-slave-v5vl krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:04.366406 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:04.346499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:56165 (local address 127.17.133.254:35507)
0925 14:52:04.346944 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:04.346950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:04.348091 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 14:52:04.350303 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:04.350309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:04.350393 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:04.350506 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:04.352495 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:04.353334 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:04.355664 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:04.356342 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:04.356476 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:04.359816 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:04.359825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:04.359832 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:04.359871 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:04.363449 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:04.364076 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:04.364084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:04.364090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:04.364226 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:04.364648 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:04.364653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:04.364656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:04.364998 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:04.365168 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:04.365810 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 14:52:04.366037 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":213,"threads_started":1}
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:52:04.367033 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:04.347083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:35507 (local address 127.17.133.193:56165)
0925 14:52:04.347909 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:04.347976 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:04.350022 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:04.350763 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:04.350774 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:04.351484 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:04.352319 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:04.352346 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:04.353486 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:04.353506 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:04.355498 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:04.355521 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:04.356538 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:04.356542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:04.356803 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:04.357444 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:04.357468 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:04.359613 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:04.363585 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:04.363591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:04.363601 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:04.363972 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:04.364346 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:04.364349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:04.364351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:04.364524 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:04.365121 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:04.365126 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:04.365387 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 14:52:04.365658 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 14:52:04.365985 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":289,"threads_started":1}
I20260925 14:52:04.368443 18706 heartbeater.cc:347] Connected to a master server at 127.17.133.254:35507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
I20260925 14:52:04.368798 18706 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:04.369680 18706 heartbeater.cc:511] Master 127.17.133.254:35507 requested a full tablet report, sending...
I20260925 14:52:04.372026 18517 ts_manager.cc:194] Registered new tserver with Master: 3caba136d0de4d239080dd750c809c3d (127.17.133.193:34443)
I20260925 14:52:04.373889 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:56165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:52:04.389468 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:35507
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:04.693099 18712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:04.693414 18712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:04.693576 18712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:04.700443 18712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:04.700579 18712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:04.700776 18712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:04.709913 18712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:35507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:04.711193 18712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:04.712548 18712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:04.725466 18717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:04.725492 18718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:04.725437 18720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:04.726608 18712 server_base.cc:1034] running on GCE node
I20260925 14:52:04.727258 18712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:04.728209 18712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:04.729444 18712 hybrid_clock.cc:648] HybridClock initialized: now 1790347924729406 us; error 55 us; skew 500 ppm
I20260925 14:52:04.729794 18712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:04 dist-test-slave-v5vl krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:04.732930 18712 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:52:04.734400 18712 webserver.cc:460] Webserver started at http://127.17.133.194:33331/ using document root <none> and password file <none>
I20260925 14:52:04.735035 18712 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:04.735132 18712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:04.735401 18712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:04.738284 18712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "ca78f0a7e1c34216aa153b7ead14d71f"
format_stamp: "Formatted at 2026-09-25 14:52:04 on dist-test-slave-v5vl"
server_key: "db13645bdfdc513cfd0a17d338998c8a"
server_key_iv: "e6100e926993398f4044a6c80a22a89b"
server_key_version: "encryptionkey@0"
I20260925 14:52:04.739022 18712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "ca78f0a7e1c34216aa153b7ead14d71f"
format_stamp: "Formatted at 2026-09-25 14:52:04 on dist-test-slave-v5vl"
server_key: "db13645bdfdc513cfd0a17d338998c8a"
server_key_iv: "e6100e926993398f4044a6c80a22a89b"
server_key_version: "encryptionkey@0"
I20260925 14:52:04.744167 18712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:52:04.747538 18727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:04.748876 18712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:04.749040 18712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "ca78f0a7e1c34216aa153b7ead14d71f"
format_stamp: "Formatted at 2026-09-25 14:52:04 on dist-test-slave-v5vl"
server_key: "db13645bdfdc513cfd0a17d338998c8a"
server_key_iv: "e6100e926993398f4044a6c80a22a89b"
server_key_version: "encryptionkey@0"
I20260925 14:52:04.749223 18712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:04.781832 18712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:04.785998 18712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:04.786291 18712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:04.787241 18712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:04.788529 18712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:04.788650 18712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:04.788785 18712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:04.788878 18712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:04.820144 18712 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:43239
I20260925 14:52:04.820188 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:43239 every 8 connection(s)
I20260925 14:52:04.821441 18712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:04.824515 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 18712
I20260925 14:52:04.824761 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:04.825353 17943 external_mini_cluster.cc:1428] Setting key f1394e71f5f67b16d7203df912b3a6a0
Sep 25 14:52:04 dist-test-slave-v5vl krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
I20260925 14:52:04.841661 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:04.823848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:59671 (local address 127.17.133.254:35507)
0925 14:52:04.824058 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:04.824064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:04.825493 (+ 1429us) server_negotiation.cc:407] Connection header received
0925 14:52:04.827635 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:04.827639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:04.827696 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:04.827778 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:04.829856 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:04.830598 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:04.832205 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:04.832692 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:04.832850 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:04.836295 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:04.836299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:04.836303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:04.836338 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:04.838841 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:04.839483 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:04.839489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:04.839495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:04.839613 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:04.840021 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:04.840026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:04.840029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:04.840280 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:04.840464 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:04.841214 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 14:52:04.841399 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 14:52:04.842553 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:04.824386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:35507 (local address 127.17.133.194:59671)
0925 14:52:04.825226 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:04.825308 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:04.827353 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:04.828048 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:04.828062 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:04.828872 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:04.829693 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:04.829723 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:04.830770 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:04.830774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:04.832045 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:04.832057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:04.832943 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:04.832950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:04.833155 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:04.833828 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:04.833848 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:04.836030 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:04.838975 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:04.838983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:04.838998 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:04.839370 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:04.839735 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:04.839740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:04.839743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:04.839913 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:04.840430 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:04.840437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:04.840802 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 14:52:04.841078 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 14:52:04.841424 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":368,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
I20260925 14:52:04.844122 18840 heartbeater.cc:347] Connected to a master server at 127.17.133.254:35507
I20260925 14:52:04.844414 18840 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:04.845148 18840 heartbeater.cc:511] Master 127.17.133.254:35507 requested a full tablet report, sending...
I20260925 14:52:04.846632 18517 ts_manager.cc:194] Registered new tserver with Master: ca78f0a7e1c34216aa153b7ead14d71f (127.17.133.194:43239)
I20260925 14:52:04.847452 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:59671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:52:04.863709 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:35507
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:05.169502 18845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:05.169806 18845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:05.169973 18845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:05.177134 18845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:05.177275 18845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:05.177455 18845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:05.186929 18845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:35507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:05.188215 18845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:05.189579 18845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:05.202418 18850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:05.202386 18853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:05.202394 18851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:05.203280 18845 server_base.cc:1034] running on GCE node
I20260925 14:52:05.203900 18845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:05.204859 18845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:05.206046 18845 hybrid_clock.cc:648] HybridClock initialized: now 1790347925206009 us; error 35 us; skew 500 ppm
I20260925 14:52:05.206347 18845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:05 dist-test-slave-v5vl krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347925, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:05.209348 18845 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:52:05.210644 18845 webserver.cc:460] Webserver started at http://127.17.133.195:37845/ using document root <none> and password file <none>
I20260925 14:52:05.211338 18845 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:05.211483 18845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:05.211814 18845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:05.214566 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "87d4b5288c2a4bf5a193600fcea17b08"
format_stamp: "Formatted at 2026-09-25 14:52:05 on dist-test-slave-v5vl"
server_key: "b15890496735aa749349590bfa15ea6f"
server_key_iv: "f8f915da60cef472fe962fa72e6e41a9"
server_key_version: "encryptionkey@0"
I20260925 14:52:05.215304 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "87d4b5288c2a4bf5a193600fcea17b08"
format_stamp: "Formatted at 2026-09-25 14:52:05 on dist-test-slave-v5vl"
server_key: "b15890496735aa749349590bfa15ea6f"
server_key_iv: "f8f915da60cef472fe962fa72e6e41a9"
server_key_version: "encryptionkey@0"
I20260925 14:52:05.219724 18845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 14:52:05.222790 18860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:05.223929 18845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:05.224094 18845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "87d4b5288c2a4bf5a193600fcea17b08"
format_stamp: "Formatted at 2026-09-25 14:52:05 on dist-test-slave-v5vl"
server_key: "b15890496735aa749349590bfa15ea6f"
server_key_iv: "f8f915da60cef472fe962fa72e6e41a9"
server_key_version: "encryptionkey@0"
I20260925 14:52:05.224284 18845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:05.243999 18845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:05.247715 18845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:05.248013 18845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:05.248958 18845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:05.250176 18845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:05.250311 18845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:05.250465 18845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:05.250537 18845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:05.282176 18845 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:41283
I20260925 14:52:05.282225 18972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:41283 every 8 connection(s)
I20260925 14:52:05.283350 18845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:05.288224 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 18845
I20260925 14:52:05.288421 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:05.288976 17943 external_mini_cluster.cc:1428] Setting key 9b72ba634d1f805eb9637321d03fc045
Sep 25 14:52:05 dist-test-slave-v5vl krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347925, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:05.301347 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.285661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:51391 (local address 127.17.133.254:35507)
0925 14:52:05.285849 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.285855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.286883 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 14:52:05.288717 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.288724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.288799 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.288906 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:05.290587 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.291520 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.293096 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.293465 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.293556 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.296402 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:05.296407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:05.296410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:05.296438 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:05.298947 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.299529 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.299533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.299535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.299638 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.299965 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.299968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.299969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.300198 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:05.300314 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.300958 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.301109 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:52:05.302176 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.286114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:35507 (local address 127.17.133.195:51391)
0925 14:52:05.286743 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:05.286777 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:05.288408 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:05.289140 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:05.289147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:05.289663 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:05.290432 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.290451 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.291699 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.291713 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.292975 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.292985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.293632 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.293636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.293812 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.294362 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:05.294385 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:05.296240 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:05.299055 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:05.299061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:05.299071 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:05.299426 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:05.299728 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:05.299731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:05.299733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:05.299882 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:05.300311 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:05.300316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:05.300567 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 14:52:05.300857 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 14:52:05.301176 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":252,"threads_started":1}
I20260925 14:52:05.303382 18973 heartbeater.cc:347] Connected to a master server at 127.17.133.254:35507
I20260925 14:52:05.303664 18973 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:05.304365 18973 heartbeater.cc:511] Master 127.17.133.254:35507 requested a full tablet report, sending...
I20260925 14:52:05.305606 18517 ts_manager.cc:194] Registered new tserver with Master: 87d4b5288c2a4bf5a193600fcea17b08 (127.17.133.195:41283)
I20260925 14:52:05.306279 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:51391
I20260925 14:52:05.315889 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:05.333706 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.321840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.17.133.254:35507)
0925 14:52:05.321995 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.322002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.322353 (+ 351us) server_negotiation.cc:407] Connection header received
0925 14:52:05.322518 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.322523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.322588 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.322699 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:05.324151 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.324926 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.326079 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.326468 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.326589 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.328886 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:05.328890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:05.328893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:05.328918 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:05.331278 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.331868 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.331873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.331877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.331995 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.332331 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.332336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.332340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.332677 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:05.332907 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.333230 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.333430 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:52:05.339143 18517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:05.346053 18517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:05.367262 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.357109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.17.133.194:43239)
0925 14:52:05.358522 (+ 1413us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.358531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.359757 (+ 1226us) server_negotiation.cc:407] Connection header received
0925 14:52:05.360049 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.360059 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.360322 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.360507 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:05.361123 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.362757 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.364798 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.366126 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.366286 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.366444 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.366757 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.366905 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269,"thread_start_us":1151,"threads_started":1}
I20260925 14:52:05.367300 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.356950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:43239 (local address 127.0.0.1:58496)
0925 14:52:05.359615 (+ 2665us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:05.359653 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:05.359842 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:05.360579 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:05.360586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:05.360637 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:05.360949 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.360966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.362889 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.362896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.364674 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.364684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.366346 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.366352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.366521 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.366570 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:52:05.366714 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 14:52:05.366877 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":22,"thread_start_us":325,"threads_started":1}
I20260925 14:52:05.368559 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.357517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:41283 (local address 127.0.0.1:36432)
0925 14:52:05.359804 (+ 2287us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:05.359826 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:05.359974 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:05.361083 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:05.361088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:05.361109 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:05.361501 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.361513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.363119 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.363124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.365819 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.365833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.367729 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.367736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.367938 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.367969 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 14:52:05.368067 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 14:52:05.368186 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2153,"mutex_wait_us":5,"thread_start_us":2594,"threads_started":1}
I20260925 14:52:05.368685 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.358113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.17.133.193:34443)
0925 14:52:05.359066 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.359077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.360476 (+ 1399us) server_negotiation.cc:407] Connection header received
0925 14:52:05.360658 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.360667 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.360903 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.361095 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:05.361931 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.363335 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.366269 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.367654 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.367867 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.368012 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.368229 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.368365 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":623,"threads_started":1}
I20260925 14:52:05.368878 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.359007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:34443 (local address 127.0.0.1:46320)
0925 14:52:05.360368 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:05.360383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:05.360493 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:05.361321 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:05.361328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:05.361355 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:05.361768 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.361778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.363530 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.363539 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.366118 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.366126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.367868 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.367874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.368029 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.368067 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:52:05.368176 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:52:05.368293 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"mutex_wait_us":54,"thread_start_us":548,"threads_started":1}
I20260925 14:52:05.369573 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.358385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.17.133.195:41283)
0925 14:52:05.359518 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.359524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.359937 (+ 413us) server_negotiation.cc:407] Connection header received
0925 14:52:05.360714 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.360722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.360916 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.361083 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:05.361738 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.362990 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.365965 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.367265 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.367466 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.367608 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.368784 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.369255 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":866,"threads_started":1}
I20260925 14:52:05.371114 18775 tablet_service.cc:1467] Processing CreateTablet for tablet e6c2075a94d04534ab365d9980efff3a (DEFAULT_TABLE table=test-table [id=ed0ef8766374483f919d2b9808a9e829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:05.371646 18641 tablet_service.cc:1467] Processing CreateTablet for tablet e6c2075a94d04534ab365d9980efff3a (DEFAULT_TABLE table=test-table [id=ed0ef8766374483f919d2b9808a9e829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:05.372196 18775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c2075a94d04534ab365d9980efff3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:05.372666 18641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c2075a94d04534ab365d9980efff3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:05.375845 18908 tablet_service.cc:1467] Processing CreateTablet for tablet e6c2075a94d04534ab365d9980efff3a (DEFAULT_TABLE table=test-table [id=ed0ef8766374483f919d2b9808a9e829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:05.376324 18706 heartbeater.cc:503] Master 127.17.133.254:35507 was elected leader, sending a full tablet report...
I20260925 14:52:05.377274 18908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c2075a94d04534ab365d9980efff3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:05.384758 18991 tablet_bootstrap.cc:492] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d: Bootstrap starting.
I20260925 14:52:05.388037 18991 tablet_bootstrap.cc:654] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:05.389205 18991 log.cc:824] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:05.389271 18992 tablet_bootstrap.cc:492] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f: Bootstrap starting.
I20260925 14:52:05.391883 18994 tablet_bootstrap.cc:492] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08: Bootstrap starting.
I20260925 14:52:05.392997 18991 tablet_bootstrap.cc:492] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d: No bootstrap required, opened a new log
I20260925 14:52:05.393221 18992 tablet_bootstrap.cc:654] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:05.393265 18991 ts_tablet_manager.cc:1397] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 14:52:05.394433 18992 log.cc:824] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:05.394724 18994 tablet_bootstrap.cc:654] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:05.395916 18994 log.cc:824] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:05.397663 18992 tablet_bootstrap.cc:492] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f: No bootstrap required, opened a new log
I20260925 14:52:05.398021 18992 ts_tablet_manager.cc:1397] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 14:52:05.402151 18991 raft_consensus.cc:348] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } }
I20260925 14:52:05.402477 18991 raft_consensus.cc:374] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:05.402599 18991 raft_consensus.cc:729] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3caba136d0de4d239080dd750c809c3d, State: Initialized, Role: FOLLOWER
I20260925 14:52:05.403028 18994 tablet_bootstrap.cc:492] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08: No bootstrap required, opened a new log
I20260925 14:52:05.403264 18994 ts_tablet_manager.cc:1397] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260925 14:52:05.403244 18991 consensus_queue.cc:260] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } }
I20260925 14:52:05.404984 18991 ts_tablet_manager.cc:1428] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260925 14:52:05.409050 18992 raft_consensus.cc:348] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } }
I20260925 14:52:05.409518 18992 raft_consensus.cc:374] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:05.409619 18992 raft_consensus.cc:729] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca78f0a7e1c34216aa153b7ead14d71f, State: Initialized, Role: FOLLOWER
I20260925 14:52:05.410213 18992 consensus_queue.cc:260] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } }
I20260925 14:52:05.411298 18840 heartbeater.cc:503] Master 127.17.133.254:35507 was elected leader, sending a full tablet report...
I20260925 14:52:05.411949 18992 ts_tablet_manager.cc:1428] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 14:52:05.413655 18994 raft_consensus.cc:348] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } }
I20260925 14:52:05.413973 18994 raft_consensus.cc:374] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:05.414069 18994 raft_consensus.cc:729] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d4b5288c2a4bf5a193600fcea17b08, State: Initialized, Role: FOLLOWER
I20260925 14:52:05.414649 18994 consensus_queue.cc:260] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } }
I20260925 14:52:05.416219 18973 heartbeater.cc:503] Master 127.17.133.254:35507 was elected leader, sending a full tablet report...
I20260925 14:52:05.416819 18994 ts_tablet_manager.cc:1428] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08: Time spent starting tablet: real 0.013s user 0.002s sys 0.013s
W20260925 14:52:05.534861 18974 tablet.cc:2369] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:05.573489 18841 tablet.cc:2369] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:05.596248 18707 tablet.cc:2369] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:05.659904 18997 raft_consensus.cc:482] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:05.660146 18997 raft_consensus.cc:504] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } }
I20260925 14:52:05.661782 18997 leader_election.cc:288] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca78f0a7e1c34216aa153b7ead14d71f (127.17.133.194:43239), 87d4b5288c2a4bf5a193600fcea17b08 (127.17.133.195:41283)
I20260925 14:52:05.669562 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.662289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:50421 (local address 127.17.133.195:41283)
0925 14:52:05.662528 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.662534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.663012 (+ 478us) server_negotiation.cc:407] Connection header received
0925 14:52:05.663215 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.663219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.663275 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.663366 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:05.663921 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.665099 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.667571 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.668309 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.668422 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.668471 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.669170 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.669305 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 14:52:05.669593 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.662053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:40205 (local address 127.17.133.194:43239)
0925 14:52:05.662273 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.662280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.663090 (+ 810us) server_negotiation.cc:407] Connection header received
0925 14:52:05.663284 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.663288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.663344 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.663439 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:05.664042 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.665167 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.667529 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.668762 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.668910 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.668977 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.669245 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.669370 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 14:52:05.669690 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.662026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:41283 (local address 127.17.133.193:50421)
0925 14:52:05.662885 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:05.662909 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:05.663059 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:05.663386 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:05.663390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:05.663413 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:05.663772 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.663784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.665264 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.665269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.667394 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.667405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.668802 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.668811 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.668999 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.669033 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:52:05.669117 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:52:05.669247 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":98,"thread_start_us":478,"threads_started":1}
I20260925 14:52:05.669718 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.662161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:43239 (local address 127.17.133.193:40205)
0925 14:52:05.662971 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:05.662984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:05.663144 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:05.663467 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:05.663472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:05.663507 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:05.663892 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.663900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.665363 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.665370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.667374 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:05.667388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.668952 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.668957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:05.669062 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.669083 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 14:52:05.669201 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 14:52:05.669312 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":25,"thread_start_us":538,"threads_started":1}
I20260925 14:52:05.670728 18795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c2075a94d04534ab365d9980efff3a" candidate_uuid: "3caba136d0de4d239080dd750c809c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" is_pre_election: true
I20260925 14:52:05.670728 18928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c2075a94d04534ab365d9980efff3a" candidate_uuid: "3caba136d0de4d239080dd750c809c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d4b5288c2a4bf5a193600fcea17b08" is_pre_election: true
I20260925 14:52:05.671200 18928 raft_consensus.cc:2454] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3caba136d0de4d239080dd750c809c3d in term 0.
I20260925 14:52:05.671221 18795 raft_consensus.cc:2454] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3caba136d0de4d239080dd750c809c3d in term 0.
I20260925 14:52:05.671809 18597 leader_election.cc:302] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3caba136d0de4d239080dd750c809c3d, 87d4b5288c2a4bf5a193600fcea17b08; no voters:
I20260925 14:52:05.672194 18997 raft_consensus.cc:2781] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:05.672340 18997 raft_consensus.cc:482] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:05.672452 18997 raft_consensus.cc:3037] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:05.674608 18997 raft_consensus.cc:504] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } }
I20260925 14:52:05.675159 18997 leader_election.cc:288] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [CANDIDATE]: Term 1 election: Requested vote from peers ca78f0a7e1c34216aa153b7ead14d71f (127.17.133.194:43239), 87d4b5288c2a4bf5a193600fcea17b08 (127.17.133.195:41283)
I20260925 14:52:05.675513 18795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c2075a94d04534ab365d9980efff3a" candidate_uuid: "3caba136d0de4d239080dd750c809c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca78f0a7e1c34216aa153b7ead14d71f"
I20260925 14:52:05.675567 18928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c2075a94d04534ab365d9980efff3a" candidate_uuid: "3caba136d0de4d239080dd750c809c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d4b5288c2a4bf5a193600fcea17b08"
I20260925 14:52:05.675719 18795 raft_consensus.cc:3037] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:05.675745 18928 raft_consensus.cc:3037] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:05.678799 18928 raft_consensus.cc:2454] T e6c2075a94d04534ab365d9980efff3a P 87d4b5288c2a4bf5a193600fcea17b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3caba136d0de4d239080dd750c809c3d in term 1.
I20260925 14:52:05.678799 18795 raft_consensus.cc:2454] T e6c2075a94d04534ab365d9980efff3a P ca78f0a7e1c34216aa153b7ead14d71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3caba136d0de4d239080dd750c809c3d in term 1.
I20260925 14:52:05.679174 18597 leader_election.cc:302] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3caba136d0de4d239080dd750c809c3d, 87d4b5288c2a4bf5a193600fcea17b08; no voters:
I20260925 14:52:05.679425 18997 raft_consensus.cc:2781] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:05.679857 18997 raft_consensus.cc:686] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [term 1 LEADER]: Becoming Leader. State: Replica: 3caba136d0de4d239080dd750c809c3d, State: Running, Role: LEADER
I20260925 14:52:05.680299 18997 consensus_queue.cc:237] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } }
I20260925 14:52:05.684466 18517 catalog_manager.cc:5495] T e6c2075a94d04534ab365d9980efff3a P 3caba136d0de4d239080dd750c809c3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3caba136d0de4d239080dd750c809c3d (127.17.133.193). New cstate: current_term: 1 leader_uuid: "3caba136d0de4d239080dd750c809c3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca78f0a7e1c34216aa153b7ead14d71f" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87d4b5288c2a4bf5a193600fcea17b08" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 41283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3caba136d0de4d239080dd750c809c3d" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34443 } health_report { overall_health: HEALTHY } } }
Sep 25 14:52:05 dist-test-slave-v5vl krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:05.737370 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.722419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.17.133.193:34443)
0925 14:52:05.722596 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.722600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.723051 (+ 451us) server_negotiation.cc:407] Connection header received
0925 14:52:05.723273 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.723283 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.723346 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.723457 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:05.724965 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.725660 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.726880 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.727393 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.727520 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.731538 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:05.731553 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:05.731569 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:05.731608 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:05.734648 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.735316 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.735322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.735325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.735469 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.735829 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.735834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.735838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.736175 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:05.736462 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.736801 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.737052 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 14:52:05 dist-test-slave-v5vl krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:52:05 dist-test-slave-v5vl krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:05.782047 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.767786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.17.133.193:34443)
0925 14:52:05.768011 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.768019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.768328 (+ 309us) server_negotiation.cc:407] Connection header received
0925 14:52:05.768545 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.768550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.768671 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.768775 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:05.770099 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.770821 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.772049 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.772462 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.772587 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.776348 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:05.776354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:05.776359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:05.776401 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:05.779771 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.780311 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.780316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.780320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.780466 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.780848 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.780853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.780857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.781153 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:05.781411 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.781543 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.781732 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 14:52:05 dist-test-slave-v5vl krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:52:05 dist-test-slave-v5vl krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:05.824409 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.810514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.17.133.193:34443)
0925 14:52:05.810731 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.810740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.811011 (+ 271us) server_negotiation.cc:407] Connection header received
0925 14:52:05.811258 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.811262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.811352 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.811458 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:05.812870 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.813631 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.814808 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.815202 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.815325 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.819089 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:05.819093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:05.819096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:05.819124 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:05.821982 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.822688 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.822692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.822694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.822822 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.823194 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.823197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.823199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.823441 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:05.823628 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.823966 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.824128 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 14:52:05.826207 18621 tablet_service.cc:2908] Rejecting scan request for tablet e6c2075a94d04534ab365d9980efff3a: Uninitialized: clean time has not yet been initialized
I20260925 14:52:05.829216 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 18578
I20260925 14:52:05.841591 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.830210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.17.133.193:34443)
0925 14:52:05.830385 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.830390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.830679 (+ 289us) server_negotiation.cc:407] Connection header received
0925 14:52:05.830806 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.830811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.830908 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.831009 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:05.832116 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.833204 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.834410 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.834999 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.835105 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.837129 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:05.837134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:05.837137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:05.837163 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:05.839416 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.839974 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.839977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.839979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.840111 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.840426 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.840430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.840432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.840747 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:05.840920 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.841153 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.841332 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:52:05.842267 18701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:05.931766 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 18578
I20260925 14:52:05.948045 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 18712
Sep 25 14:52:05 dist-test-slave-v5vl krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:52:05.961724 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:05.948791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.17.133.194:43239)
0925 14:52:05.948954 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:05.948961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:05.948983 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:52:05.949201 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:05.949206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:05.949275 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:05.949374 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:05.950465 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.951171 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.952458 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:05.952935 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:05.953072 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:05.956539 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:05.956553 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:05.956574 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:05.956643 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:05.959075 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.959716 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.959724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.959728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.959854 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:05.960202 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:05.960208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:05.960211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:05.960571 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:05.960898 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:05.961194 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 14:52:05.961401 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:52:05.962325 18835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:06.046227 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 18712
I20260925 14:52:06.062191 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 18845
Sep 25 14:52:06 dist-test-slave-v5vl krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:52:06.075889 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:06.062936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.17.133.195:41283)
0925 14:52:06.063093 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:06.063097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:06.063115 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:52:06.063247 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:06.063252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:06.063311 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:06.063393 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:06.064514 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:06.065304 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:06.066514 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:06.066882 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:06.066981 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:06.070895 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:06.070909 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:06.070925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:06.070956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:06.073721 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:06.074261 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:06.074268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:06.074270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:06.074392 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:06.074682 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:06.074686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:06.074687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:06.074999 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:06.075194 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:06.075478 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 14:52:06.075626 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:52:06.076499 18968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:06.160332 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 18845
I20260925 14:52:06.177485 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 18485
I20260925 14:52:06.178779 18547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:06.292894 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 18485
2026-09-25T14:52:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4160 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:06 dist-test-slave-v5vl krb5kdc[19021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:06 dist-test-slave-v5vl krb5kdc[19021](info): set up 2 sockets
Sep 25 14:52:06 dist-test-slave-v5vl krb5kdc[19021](info): commencing operation
krb5kdc: starting...
W20260925 14:52:07.361323 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:07 dist-test-slave-v5vl krb5kdc[19021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:52:07.525679 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:41161
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:43923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:41161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:07.834909 19034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:07.835302 19034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:07.835533 19034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:07.842871 19034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:07.842967 19034 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:52:07.843041 19034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:07.843134 19034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:07.843221 19034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:07.843312 19034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:07.853320 19034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:43923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:41161
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:07.854588 19034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:07.855921 19034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:07.867204 19039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:07.867255 19040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:07.867281 19042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:07.868191 19034 server_base.cc:1034] running on GCE node
I20260925 14:52:07.868961 19034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:07.870307 19034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:07.871532 19034 hybrid_clock.cc:648] HybridClock initialized: now 1790347927871463 us; error 42 us; skew 500 ppm
I20260925 14:52:07.871845 19034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:07 dist-test-slave-v5vl krb5kdc[19021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:07.874806 19034 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:52:07.876118 19034 webserver.cc:460] Webserver started at http://127.17.133.254:33121/ using document root <none> and password file <none>
I20260925 14:52:07.876865 19034 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:07.876991 19034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:07.877316 19034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:07.880043 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "e91df70185c04666ab70422ede015714"
format_stamp: "Formatted at 2026-09-25 14:52:07 on dist-test-slave-v5vl"
server_key: "8809354b568707fa737c9c62f2424988"
server_key_iv: "f3f3d7ee4c8d58041f2ff8220e558378"
server_key_version: "encryptionkey@0"
I20260925 14:52:07.880847 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "e91df70185c04666ab70422ede015714"
format_stamp: "Formatted at 2026-09-25 14:52:07 on dist-test-slave-v5vl"
server_key: "8809354b568707fa737c9c62f2424988"
server_key_iv: "f3f3d7ee4c8d58041f2ff8220e558378"
server_key_version: "encryptionkey@0"
I20260925 14:52:07.885588 19034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 14:52:07.888852 19049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:07.890072 19034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:52:07.890242 19034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "e91df70185c04666ab70422ede015714"
format_stamp: "Formatted at 2026-09-25 14:52:07 on dist-test-slave-v5vl"
server_key: "8809354b568707fa737c9c62f2424988"
server_key_iv: "f3f3d7ee4c8d58041f2ff8220e558378"
server_key_version: "encryptionkey@0"
I20260925 14:52:07.890448 19034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:07.906332 19034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:07.910614 19034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:07.910885 19034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:07.931684 19034 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:41161
I20260925 14:52:07.931701 19110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:41161 every 8 connection(s)
I20260925 14:52:07.933072 19034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:07.936980 19111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:07.941076 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 19034
I20260925 14:52:07.941296 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:07.942066 17943 external_mini_cluster.cc:1428] Setting key a2231f617cad2dd05956b648d86863a2
I20260925 14:52:07.948310 19111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714: Bootstrap starting.
I20260925 14:52:07.951737 19111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:07.952912 19111 log.cc:824] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:07.956164 19111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714: No bootstrap required, opened a new log
Sep 25 14:52:07 dist-test-slave-v5vl krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:07.963752 19111 raft_consensus.cc:348] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91df70185c04666ab70422ede015714" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41161 } }
I20260925 14:52:07.964099 19111 raft_consensus.cc:374] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:07.964222 19111 raft_consensus.cc:729] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91df70185c04666ab70422ede015714, State: Initialized, Role: FOLLOWER
I20260925 14:52:07.964773 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:07.944017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.17.133.254:41161)
0925 14:52:07.944805 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:07.944816 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:07.944857 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:52:07.946891 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:07.946913 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:07.947264 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:07.947564 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:07.949209 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.950432 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.952072 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.952842 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.952972 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:07.957685 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:07.957696 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:07.957709 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:07.957767 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:07.960866 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:07.961519 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:07.961526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:07.961534 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:07.961708 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:07.962066 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:07.962073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:07.962083 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:07.962579 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:07.962891 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:07.963267 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 14:52:07.963641 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":365,"threads_started":1}
I20260925 14:52:07.964917 19111 consensus_queue.cc:260] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91df70185c04666ab70422ede015714" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41161 } }
I20260925 14:52:07.965212 19111 raft_consensus.cc:388] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:07.965374 19111 raft_consensus.cc:482] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:07.965555 19111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:07.968497 19111 raft_consensus.cc:504] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91df70185c04666ab70422ede015714" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41161 } }
I20260925 14:52:07.969059 19111 leader_election.cc:302] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e91df70185c04666ab70422ede015714; no voters:
I20260925 14:52:07.969611 19111 leader_election.cc:288] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:07.969724 19116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:07.970832 19116 raft_consensus.cc:686] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [term 1 LEADER]: Becoming Leader. State: Replica: e91df70185c04666ab70422ede015714, State: Running, Role: LEADER
I20260925 14:52:07.971397 19116 consensus_queue.cc:237] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91df70185c04666ab70422ede015714" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41161 } }
I20260925 14:52:07.971709 19111 sys_catalog.cc:564] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:07.973907 19117 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e91df70185c04666ab70422ede015714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91df70185c04666ab70422ede015714" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41161 } } }
I20260925 14:52:07.974637 19117 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:07.974421 19118 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e91df70185c04666ab70422ede015714. Latest consensus state: current_term: 1 leader_uuid: "e91df70185c04666ab70422ede015714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91df70185c04666ab70422ede015714" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41161 } } }
I20260925 14:52:07.974923 19118 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:07.978020 19121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:07.983471 19121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:07.991573 19121 catalog_manager.cc:1347] Generated new cluster ID: 889d9178f1cd4cc59e7182938198a6f7
I20260925 14:52:07.991679 19121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:08.006669 19121 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:08.007944 19121 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:08.014829 19121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714: Generated new TSK 0
I20260925 14:52:08.015743 19121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:52:08.062377 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:41161
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:43923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:08.404284 19137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:08.404551 19137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:08.404768 19137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:08.411864 19137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:08.411981 19137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:08.412129 19137 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:52:08.412249 19137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:08.421362 19137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:43923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.133.254:41161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:08.422617 19137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:08.423885 19137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:08.436599 19142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:08.436506 19145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:08.436728 19143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:08.437328 19137 server_base.cc:1034] running on GCE node
I20260925 14:52:08.437952 19137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:08.438800 19137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:08.440045 19137 hybrid_clock.cc:648] HybridClock initialized: now 1790347928439981 us; error 39 us; skew 500 ppm
I20260925 14:52:08.440333 19137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:08 dist-test-slave-v5vl krb5kdc[19021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:08.443397 19137 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:52:08.444790 19137 webserver.cc:460] Webserver started at http://127.17.133.193:38655/ using document root <none> and password file <none>
I20260925 14:52:08.445449 19137 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:08.445576 19137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:08.445879 19137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:08.448705 19137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "71df639daaff42e2a8cc0e38e2a92350"
format_stamp: "Formatted at 2026-09-25 14:52:08 on dist-test-slave-v5vl"
server_key: "2a4a585a6d44152efb43d8bb9c0ecd4d"
server_key_iv: "5cf13a05fbb40efb9242ec424dfb3ab6"
server_key_version: "encryptionkey@0"
I20260925 14:52:08.449470 19137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "71df639daaff42e2a8cc0e38e2a92350"
format_stamp: "Formatted at 2026-09-25 14:52:08 on dist-test-slave-v5vl"
server_key: "2a4a585a6d44152efb43d8bb9c0ecd4d"
server_key_iv: "5cf13a05fbb40efb9242ec424dfb3ab6"
server_key_version: "encryptionkey@0"
I20260925 14:52:08.454303 19137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 14:52:08.457537 19152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:08.458719 19137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:52:08.458887 19137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "71df639daaff42e2a8cc0e38e2a92350"
format_stamp: "Formatted at 2026-09-25 14:52:08 on dist-test-slave-v5vl"
server_key: "2a4a585a6d44152efb43d8bb9c0ecd4d"
server_key_iv: "5cf13a05fbb40efb9242ec424dfb3ab6"
server_key_version: "encryptionkey@0"
I20260925 14:52:08.459105 19137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:08.483989 19137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:08.488373 19137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:08.488711 19137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:08.491307 19137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:08.491416 19137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:08.491572 19137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:08.491652 19137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:52:08 dist-test-slave-v5vl krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:08.516423 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.491972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:41327 (local address 127.17.133.254:41161)
0925 14:52:08.494799 (+ 2827us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.494818 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.494844 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:52:08.497158 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.497162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.497230 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.497313 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:08.499748 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.500767 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.504140 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.504689 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.504790 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.509384 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:08.509393 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:08.509401 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:08.509453 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:08.513332 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:08.514034 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:08.514043 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:08.514050 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:08.514202 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:08.514628 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:08.514636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:08.514643 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:08.514946 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:08.515108 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.515864 (+ 756us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.516066 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2696,"thread_start_us":2607,"threads_started":1}
I20260925 14:52:08.517596 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.492809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41161 (local address 127.17.133.193:41327)
0925 14:52:08.493943 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.494002 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.496808 (+ 2806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.497621 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:08.497635 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:08.498489 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:08.499525 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.499562 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.500974 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.500990 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.503956 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.503974 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.504927 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.504937 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.505334 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.506242 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:08.506265 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:08.509100 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:08.513469 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:08.513475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:08.513487 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:08.513914 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:08.514334 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:08.514338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:08.514340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:08.514510 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:08.515195 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:08.515201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:08.515517 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 14:52:08.515793 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 14:52:08.516327 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":444,"threads_started":1}
I20260925 14:52:08.538905 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.523210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:48771 (local address 127.17.133.254:41161)
0925 14:52:08.523383 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.523391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.524222 (+ 831us) server_negotiation.cc:407] Connection header received
0925 14:52:08.524440 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.524444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.524521 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.524661 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:08.526387 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.527321 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.529529 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.530066 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.530194 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.532823 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:08.532828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:08.532831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:08.532888 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:08.535823 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:08.536787 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:08.536792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:08.536795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:08.536976 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:08.537340 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:08.537346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:08.537349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:08.537605 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:08.537767 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.538340 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.538564 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:52:08.539029 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.523311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41161 (local address 127.17.133.193:48771)
0925 14:52:08.524120 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.524135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.524294 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.524712 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:08.524718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:08.525525 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:08.526212 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.526235 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.527489 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.527498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.529320 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.529340 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.530270 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.530280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.530576 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.531349 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:08.531365 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:08.532638 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:08.535988 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:08.535992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:08.535997 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:08.536631 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:08.537085 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:08.537091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:08.537095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:08.537228 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:08.537831 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:08.537835 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:08.538059 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 14:52:08.538212 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 14:52:08.538563 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":557,"threads_started":1}
I20260925 14:52:08.549674 19137 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:43043
I20260925 14:52:08.549695 19272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:43043 every 8 connection(s)
I20260925 14:52:08.550863 19137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:08.553054 19273 heartbeater.cc:347] Connected to a master server at 127.17.133.254:41161
I20260925 14:52:08.553262 19273 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:08.553807 19273 heartbeater.cc:511] Master 127.17.133.254:41161 requested a full tablet report, sending...
I20260925 14:52:08.555586 19066 ts_manager.cc:194] Registered new tserver with Master: 71df639daaff42e2a8cc0e38e2a92350 (127.17.133.193:43043)
I20260925 14:52:08.556947 19066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:41327
I20260925 14:52:08.558830 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 19137
I20260925 14:52:08.559032 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:08.559620 17943 external_mini_cluster.cc:1428] Setting key 00607270476e3f04d169f291b624e767
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:52:08.595903 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:41161
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:43923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:08.904294 19277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:08.904572 19277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:08.904714 19277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:08.911826 19277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:08.911942 19277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:08.912060 19277 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:52:08.912225 19277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:08.921628 19277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:43923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.133.254:41161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:08.922804 19277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:08.924077 19277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:08.943357 19282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:08.943542 19283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:08.944186 19285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:08.944402 19277 server_base.cc:1034] running on GCE node
I20260925 14:52:08.945029 19277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:08.945943 19277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:08.947165 19277 hybrid_clock.cc:648] HybridClock initialized: now 1790347928947098 us; error 47 us; skew 500 ppm
I20260925 14:52:08.947448 19277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:08 dist-test-slave-v5vl krb5kdc[19021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:08.953269 19277 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:52:08.954671 19277 webserver.cc:460] Webserver started at http://127.17.133.194:34581/ using document root <none> and password file <none>
I20260925 14:52:08.955461 19277 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:08.955592 19277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:08.955926 19277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:08.958777 19277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "3cba4d073bad4b61a5801de5fbe2a32c"
format_stamp: "Formatted at 2026-09-25 14:52:08 on dist-test-slave-v5vl"
server_key: "33c8439458fc14afab49a8ec21005878"
server_key_iv: "559d6ceaf9635ba71cf5ef74ff7079f0"
server_key_version: "encryptionkey@0"
I20260925 14:52:08.959545 19277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "3cba4d073bad4b61a5801de5fbe2a32c"
format_stamp: "Formatted at 2026-09-25 14:52:08 on dist-test-slave-v5vl"
server_key: "33c8439458fc14afab49a8ec21005878"
server_key_iv: "559d6ceaf9635ba71cf5ef74ff7079f0"
server_key_version: "encryptionkey@0"
I20260925 14:52:08.964723 19277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 14:52:08.967981 19292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:08.969321 19277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:08.969498 19277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "3cba4d073bad4b61a5801de5fbe2a32c"
format_stamp: "Formatted at 2026-09-25 14:52:08 on dist-test-slave-v5vl"
server_key: "33c8439458fc14afab49a8ec21005878"
server_key_iv: "559d6ceaf9635ba71cf5ef74ff7079f0"
server_key_version: "encryptionkey@0"
I20260925 14:52:08.969709 19277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:08.982836 19277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:08.987529 19277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:08.987807 19277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:08.990399 19277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:08.990501 19277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:08.990640 19277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:08.990713 19277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:52:09 dist-test-slave-v5vl krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:09.015487 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.991052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:45945 (local address 127.17.133.254:41161)
0925 14:52:08.992312 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.992318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.992774 (+ 456us) server_negotiation.cc:407] Connection header received
0925 14:52:08.995172 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.995180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.995281 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.995405 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:08.998134 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.999079 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.002514 (+ 3435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.003078 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.003225 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.008143 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:09.008151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:09.008160 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:09.008193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:09.011406 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.012280 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.012289 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.012295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.012462 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.013108 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.013115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.013121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.013509 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:09.013667 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.014802 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.015036 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120}
I20260925 14:52:09.017006 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.991793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41161 (local address 127.17.133.194:45945)
0925 14:52:08.992570 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.992655 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.994825 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.995847 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:08.995866 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:08.996881 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:08.997920 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.997945 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.999265 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.999273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.002227 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.002270 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.003600 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.003610 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.003958 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.004770 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:09.004795 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:09.007909 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:09.011549 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:09.011557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:09.011573 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:09.012149 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:09.012613 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:09.012620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:09.012625 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:09.012947 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:09.013680 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:09.013694 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:09.014246 (+ 552us) client_negotiation.cc:769] Sending connection context
0925 14:52:09.014643 (+ 397us) client_negotiation.cc:240] Negotiation successful
0925 14:52:09.015173 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":418,"threads_started":1}
I20260925 14:52:09.036703 19277 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:34615
I20260925 14:52:09.036741 19411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:34615 every 8 connection(s)
I20260925 14:52:09.038033 19277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:09.046226 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 19277
I20260925 14:52:09.046412 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:09.046938 17943 external_mini_cluster.cc:1428] Setting key 19e269be72d63e85816382c60b2a7252
I20260925 14:52:09.050401 19412 heartbeater.cc:347] Connected to a master server at 127.17.133.254:41161
I20260925 14:52:09.050487 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.024436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:34481 (local address 127.17.133.254:41161)
0925 14:52:09.024667 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.024675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.029210 (+ 4535us) server_negotiation.cc:407] Connection header received
0925 14:52:09.029341 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.029349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.029443 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.029573 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:09.030363 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.031771 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.034267 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.034805 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.034935 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.037141 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:09.037147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:09.037152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:09.037188 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:09.040433 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.047606 (+ 7173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.047613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.047615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.047799 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.048829 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.048837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.048841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.049365 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:09.049521 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.049941 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.050143 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:52:09.050632 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.024438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41161 (local address 127.17.133.194:34481)
0925 14:52:09.029083 (+ 4645us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:09.029107 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:09.029231 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:09.029583 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:09.029586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:09.029953 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:09.030211 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.030223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.031976 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.031984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.034141 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.034154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.035034 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.035044 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.035302 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.036042 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:09.036059 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:09.036975 (+ 916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:09.040544 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:09.040551 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:09.040556 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:09.041191 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:09.047918 (+ 6727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:09.047923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:09.047927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:09.048088 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:09.049509 (+ 1421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:09.049514 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:09.049691 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 14:52:09.049826 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 14:52:09.050137 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4500,"thread_start_us":4370,"threads_started":1}
I20260925 14:52:09.050683 19412 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:09.051525 19412 heartbeater.cc:511] Master 127.17.133.254:41161 requested a full tablet report, sending...
I20260925 14:52:09.052997 19066 ts_manager.cc:194] Registered new tserver with Master: 3cba4d073bad4b61a5801de5fbe2a32c (127.17.133.194:34615)
I20260925 14:52:09.054086 19066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:45945
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:52:09.085367 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:41161
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:43923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:09.395913 19416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:09.396241 19416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:09.396413 19416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:09.403934 19416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:09.404094 19416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:09.404250 19416 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:52:09.404394 19416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:09.414286 19416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:43923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.133.254:41161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:09.415597 19416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:09.416917 19416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:09.429658 19421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:09.429625 19422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:09.429594 19424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:09.430377 19416 server_base.cc:1034] running on GCE node
I20260925 14:52:09.430936 19416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:09.431804 19416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:09.433037 19416 hybrid_clock.cc:648] HybridClock initialized: now 1790347929432988 us; error 49 us; skew 500 ppm
I20260925 14:52:09.433334 19416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:09 dist-test-slave-v5vl krb5kdc[19021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347929, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:09.436362 19416 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:52:09.437808 19416 webserver.cc:460] Webserver started at http://127.17.133.195:45043/ using document root <none> and password file <none>
I20260925 14:52:09.438486 19416 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:09.438612 19416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:09.438941 19416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:09.441870 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "5df5f86546eb4c539c5ad079a369bbff"
format_stamp: "Formatted at 2026-09-25 14:52:09 on dist-test-slave-v5vl"
server_key: "4ac33e528974d7b29815861630c88383"
server_key_iv: "0f5819dc7ca5e2b842906c9c3d96a0ad"
server_key_version: "encryptionkey@0"
I20260925 14:52:09.442672 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "5df5f86546eb4c539c5ad079a369bbff"
format_stamp: "Formatted at 2026-09-25 14:52:09 on dist-test-slave-v5vl"
server_key: "4ac33e528974d7b29815861630c88383"
server_key_iv: "0f5819dc7ca5e2b842906c9c3d96a0ad"
server_key_version: "encryptionkey@0"
I20260925 14:52:09.447503 19416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 14:52:09.450747 19431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:09.451939 19416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 14:52:09.452101 19416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "5df5f86546eb4c539c5ad079a369bbff"
format_stamp: "Formatted at 2026-09-25 14:52:09 on dist-test-slave-v5vl"
server_key: "4ac33e528974d7b29815861630c88383"
server_key_iv: "0f5819dc7ca5e2b842906c9c3d96a0ad"
server_key_version: "encryptionkey@0"
I20260925 14:52:09.452297 19416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:09.470885 19416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:09.474947 19416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:09.475196 19416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:09.477854 19416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:09.477958 19416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:09.478169 19416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:09.478247 19416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:52:09 dist-test-slave-v5vl krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347929, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:09.499831 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.478286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:58121 (local address 127.17.133.254:41161)
0925 14:52:09.478533 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.478542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.479822 (+ 1280us) server_negotiation.cc:407] Connection header received
0925 14:52:09.482065 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.482071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.482157 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.482274 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:09.484422 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.485470 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.488086 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.488503 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.488659 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.493273 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:09.493281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:09.493289 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:09.493336 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:09.496073 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.497213 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.497219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.497223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.497400 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.497951 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.497957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.497960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.498227 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:09.498393 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.499260 (+ 867us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.499479 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 14:52:09.501339 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.478922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41161 (local address 127.17.133.195:58121)
0925 14:52:09.479663 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:09.479703 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:09.481726 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:09.482553 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:09.482568 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:09.483384 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:09.484249 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.484268 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.485709 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.485716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.487897 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.487926 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.488755 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.488762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.489061 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.489852 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:09.489890 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:09.493047 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:09.496220 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:09.496233 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:09.496258 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:09.497023 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:09.497544 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:09.497555 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:09.497559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:09.497815 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:09.498371 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:09.498378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:09.498789 (+ 411us) client_negotiation.cc:769] Sending connection context
0925 14:52:09.499153 (+ 364us) client_negotiation.cc:240] Negotiation successful
0925 14:52:09.499708 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":324,"threads_started":1}
I20260925 14:52:09.525110 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.508339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:51737 (local address 127.17.133.254:41161)
0925 14:52:09.508500 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.508504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.509537 (+ 1033us) server_negotiation.cc:407] Connection header received
0925 14:52:09.509725 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.509728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.509793 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.509904 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:09.511362 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.512335 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.514622 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.515147 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.515280 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.518004 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:09.518009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:09.518012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:09.518038 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:09.521995 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.522765 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.522774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.522779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.522947 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.523447 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.523452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.523455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.523805 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:09.523984 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.524485 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.524737 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:52:09.525224 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.508622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41161 (local address 127.17.133.195:51737)
0925 14:52:09.509407 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:09.509432 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:09.509606 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:09.509926 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:09.509934 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:09.510604 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:09.511230 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.511249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.512490 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.512500 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.514443 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.514475 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.515359 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.515368 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.515648 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.516308 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:09.516334 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:09.517830 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:09.522133 (+ 4303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:09.522137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:09.522139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:09.522590 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:09.523070 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:09.523081 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:09.523085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:09.523270 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:09.524047 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:09.524051 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:09.524230 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 14:52:09.524381 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 14:52:09.524740 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":427,"threads_started":1}
I20260925 14:52:09.526427 19416 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:44889
I20260925 14:52:09.526445 19550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:44889 every 8 connection(s)
I20260925 14:52:09.527714 19416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:09.530171 19551 heartbeater.cc:347] Connected to a master server at 127.17.133.254:41161
I20260925 14:52:09.530407 19551 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:09.531013 19551 heartbeater.cc:511] Master 127.17.133.254:41161 requested a full tablet report, sending...
I20260925 14:52:09.531303 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 19416
I20260925 14:52:09.531504 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:09.531966 17943 external_mini_cluster.cc:1428] Setting key 60e91478a35efd98b23fac3c1ae2a9a9
I20260925 14:52:09.532414 19066 ts_manager.cc:194] Registered new tserver with Master: 5df5f86546eb4c539c5ad079a369bbff (127.17.133.195:44889)
I20260925 14:52:09.533334 19066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:58121
I20260925 14:52:09.534834 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:52:09 dist-test-slave-v5vl krb5kdc[19021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 14:52:09.559306 19273 heartbeater.cc:503] Master 127.17.133.254:41161 was elected leader, sending a full tablet report...
Sep 25 14:52:09 dist-test-slave-v5vl krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:09.577991 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.565232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.17.133.254:41161)
0925 14:52:09.565394 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.565399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.565786 (+ 387us) server_negotiation.cc:407] Connection header received
0925 14:52:09.565923 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.565927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.565982 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.566067 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:09.567328 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.568026 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.569261 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.569664 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.569761 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.573329 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:09.573352 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:09.573355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:09.573383 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:09.575913 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.576450 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.576453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.576455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.576563 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:09.576910 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:09.576913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:09.576915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:09.577157 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:09.577340 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.577532 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.577740 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:52:09.581686 19066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:09.588109 19066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:09.610998 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.599772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.17.133.193:43043)
0925 14:52:09.602796 (+ 3024us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.602807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.602837 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:52:09.602943 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.602950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.603270 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.604286 (+ 1016us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:09.604490 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.605964 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.608164 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.609696 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.609896 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.610085 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.610393 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.610574 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2843,"thread_start_us":3360,"threads_started":1}
I20260925 14:52:09.610898 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.599568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43043 (local address 127.0.0.1:42682)
0925 14:52:09.600274 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:09.600325 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:09.600496 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:09.603523 (+ 3027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:09.603532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:09.603569 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:09.604003 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.604017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.606087 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.606095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.607978 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.607989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.609922 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.609928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.610119 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.610171 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 14:52:09.610310 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 14:52:09.610506 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":338,"threads_started":1}
I20260925 14:52:09.612322 19566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.599922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:44889 (local address 127.0.0.1:49574)
0925 14:52:09.602528 (+ 2606us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:09.602548 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:09.602653 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:09.604888 (+ 2235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:09.604895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:09.604924 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:09.605333 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.605345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.607320 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.607324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.609256 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.609267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.610951 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.610957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.611121 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.611160 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 14:52:09.611224 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:52:09.611354 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2417,"mutex_wait_us":105,"thread_start_us":2252,"threads_started":1}
I20260925 14:52:09.612632 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.602312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:34615 (local address 127.0.0.1:42222)
0925 14:52:09.603802 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:09.603823 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:09.603977 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:09.605161 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:09.605166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:09.605189 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:09.605566 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.605577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.607318 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.607323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.609757 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.609773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.611798 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.611802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.611981 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.612028 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 14:52:09.612108 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:52:09.612254 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"mutex_wait_us":685,"thread_start_us":511,"threads_started":1}
I20260925 14:52:09.612884 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.600622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.17.133.195:44889)
0925 14:52:09.603262 (+ 2640us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.603270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.603302 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:52:09.604445 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.604454 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.604697 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.604900 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:09.605526 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.607155 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.609409 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.610711 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.611518 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.611685 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.611757 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.612527 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2462,"thread_start_us":2295,"threads_started":1}
I20260925 14:52:09.613166 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.602745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.17.133.194:34615)
0925 14:52:09.604450 (+ 1705us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.604456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.604475 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:52:09.604549 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.604553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.604773 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.604982 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:09.605748 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.607155 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.609949 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.611332 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.611492 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.611665 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.612725 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.612887 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1568,"thread_start_us":1459,"threads_started":1}
I20260925 14:52:09.615895 19203 tablet_service.cc:1467] Processing CreateTablet for tablet 927ef5d5838c4c5aaaf45b1572925f35 (DEFAULT_TABLE table=test-table [id=8c94e139a3eb42aba4a5454f1e198f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:09.615806 19481 tablet_service.cc:1467] Processing CreateTablet for tablet 927ef5d5838c4c5aaaf45b1572925f35 (DEFAULT_TABLE table=test-table [id=8c94e139a3eb42aba4a5454f1e198f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:09.616323 19342 tablet_service.cc:1467] Processing CreateTablet for tablet 927ef5d5838c4c5aaaf45b1572925f35 (DEFAULT_TABLE table=test-table [id=8c94e139a3eb42aba4a5454f1e198f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:09.617283 19481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927ef5d5838c4c5aaaf45b1572925f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:09.617295 19203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927ef5d5838c4c5aaaf45b1572925f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:09.617446 19342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927ef5d5838c4c5aaaf45b1572925f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:09.631637 19570 tablet_bootstrap.cc:492] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: Bootstrap starting.
I20260925 14:52:09.634864 19570 tablet_bootstrap.cc:654] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:09.635887 19570 log.cc:824] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:09.636797 19572 tablet_bootstrap.cc:492] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: Bootstrap starting.
I20260925 14:52:09.637739 19571 tablet_bootstrap.cc:492] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: Bootstrap starting.
I20260925 14:52:09.640115 19570 tablet_bootstrap.cc:492] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: No bootstrap required, opened a new log
I20260925 14:52:09.640419 19570 ts_tablet_manager.cc:1397] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 14:52:09.641194 19572 tablet_bootstrap.cc:654] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:09.641194 19571 tablet_bootstrap.cc:654] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:09.642427 19571 log.cc:824] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:09.642428 19572 log.cc:824] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:09.645059 19572 tablet_bootstrap.cc:492] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: No bootstrap required, opened a new log
I20260925 14:52:09.645358 19572 ts_tablet_manager.cc:1397] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260925 14:52:09.645521 19571 tablet_bootstrap.cc:492] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: No bootstrap required, opened a new log
I20260925 14:52:09.645766 19571 ts_tablet_manager.cc:1397] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 14:52:09.650660 19570 raft_consensus.cc:348] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } }
I20260925 14:52:09.651072 19570 raft_consensus.cc:374] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:09.651170 19570 raft_consensus.cc:729] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df5f86546eb4c539c5ad079a369bbff, State: Initialized, Role: FOLLOWER
I20260925 14:52:09.651750 19570 consensus_queue.cc:260] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } }
I20260925 14:52:09.652838 19551 heartbeater.cc:503] Master 127.17.133.254:41161 was elected leader, sending a full tablet report...
I20260925 14:52:09.653436 19570 ts_tablet_manager.cc:1428] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260925 14:52:09.653695 19572 raft_consensus.cc:348] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } }
I20260925 14:52:09.654078 19572 raft_consensus.cc:374] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:09.654182 19572 raft_consensus.cc:729] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cba4d073bad4b61a5801de5fbe2a32c, State: Initialized, Role: FOLLOWER
I20260925 14:52:09.654695 19572 consensus_queue.cc:260] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } }
I20260925 14:52:09.655642 19412 heartbeater.cc:503] Master 127.17.133.254:41161 was elected leader, sending a full tablet report...
I20260925 14:52:09.655545 19571 raft_consensus.cc:348] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } }
I20260925 14:52:09.655911 19571 raft_consensus.cc:374] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:09.656005 19571 raft_consensus.cc:729] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df639daaff42e2a8cc0e38e2a92350, State: Initialized, Role: FOLLOWER
I20260925 14:52:09.656280 19572 ts_tablet_manager.cc:1428] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260925 14:52:09.656507 19571 consensus_queue.cc:260] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } }
I20260925 14:52:09.658097 19571 ts_tablet_manager.cc:1428] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260925 14:52:09.779671 19552 tablet.cc:2369] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:09.790519 19413 tablet.cc:2369] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:09.803035 19274 tablet.cc:2369] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:09.892051 19578 raft_consensus.cc:482] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:09.892462 19578 raft_consensus.cc:504] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } }
I20260925 14:52:09.894255 19578 leader_election.cc:288] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5df5f86546eb4c539c5ad079a369bbff (127.17.133.195:44889), 3cba4d073bad4b61a5801de5fbe2a32c (127.17.133.194:34615)
I20260925 14:52:09.900641 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.894554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:42149 (local address 127.17.133.195:44889)
0925 14:52:09.894763 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.894773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.895404 (+ 631us) server_negotiation.cc:407] Connection header received
0925 14:52:09.895624 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.895629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.895697 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.895801 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:09.896298 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.897073 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.898884 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.899614 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.899801 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.899874 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.900222 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.900373 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:52:09.900774 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.894479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:44889 (local address 127.17.133.193:42149)
0925 14:52:09.895292 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:09.895307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:09.895477 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:09.895808 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:09.895813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:09.895837 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:09.896151 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.896159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.897200 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.897208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.898725 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.898736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.899809 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.899815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.900058 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.900081 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:52:09.900197 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 14:52:09.900367 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":52,"thread_start_us":482,"threads_started":1}
I20260925 14:52:09.901105 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.894891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:53563 (local address 127.17.133.194:34615)
0925 14:52:09.895134 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.895143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.895738 (+ 595us) server_negotiation.cc:407] Connection header received
0925 14:52:09.895929 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.895935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.895997 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.896091 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:09.896676 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.897388 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.899041 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.900129 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.900329 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.900403 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.900713 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.900847 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 14:52:09.901495 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.894688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:34615 (local address 127.17.133.193:53563)
0925 14:52:09.895625 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:09.895646 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:09.895788 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:09.896122 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:09.896126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:09.896151 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:09.896497 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.896504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.897522 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.897528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.898878 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:09.898889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.900371 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.900377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:09.900530 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.900552 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:52:09.900691 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 14:52:09.901218 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":643,"threads_started":1}
I20260925 14:52:09.901700 19497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927ef5d5838c4c5aaaf45b1572925f35" candidate_uuid: "71df639daaff42e2a8cc0e38e2a92350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df5f86546eb4c539c5ad079a369bbff" is_pre_election: true
I20260925 14:52:09.902189 19497 raft_consensus.cc:2454] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df639daaff42e2a8cc0e38e2a92350 in term 0.
I20260925 14:52:09.902366 19362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927ef5d5838c4c5aaaf45b1572925f35" candidate_uuid: "71df639daaff42e2a8cc0e38e2a92350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" is_pre_election: true
I20260925 14:52:09.902858 19362 raft_consensus.cc:2454] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df639daaff42e2a8cc0e38e2a92350 in term 0.
I20260925 14:52:09.902863 19154 leader_election.cc:302] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5df5f86546eb4c539c5ad079a369bbff, 71df639daaff42e2a8cc0e38e2a92350; no voters:
I20260925 14:52:09.903266 19578 raft_consensus.cc:2781] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:09.903406 19578 raft_consensus.cc:482] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:09.903524 19578 raft_consensus.cc:3037] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:09.905732 19578 raft_consensus.cc:504] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } }
I20260925 14:52:09.906303 19578 leader_election.cc:288] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [CANDIDATE]: Term 1 election: Requested vote from peers 5df5f86546eb4c539c5ad079a369bbff (127.17.133.195:44889), 3cba4d073bad4b61a5801de5fbe2a32c (127.17.133.194:34615)
I20260925 14:52:09.906657 19497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927ef5d5838c4c5aaaf45b1572925f35" candidate_uuid: "71df639daaff42e2a8cc0e38e2a92350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df5f86546eb4c539c5ad079a369bbff"
I20260925 14:52:09.906765 19362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927ef5d5838c4c5aaaf45b1572925f35" candidate_uuid: "71df639daaff42e2a8cc0e38e2a92350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cba4d073bad4b61a5801de5fbe2a32c"
I20260925 14:52:09.906867 19497 raft_consensus.cc:3037] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:09.906944 19362 raft_consensus.cc:3037] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:09.909767 19497 raft_consensus.cc:2454] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df639daaff42e2a8cc0e38e2a92350 in term 1.
I20260925 14:52:09.909767 19362 raft_consensus.cc:2454] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df639daaff42e2a8cc0e38e2a92350 in term 1.
I20260925 14:52:09.910107 19155 leader_election.cc:302] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cba4d073bad4b61a5801de5fbe2a32c, 71df639daaff42e2a8cc0e38e2a92350; no voters:
I20260925 14:52:09.910825 19578 raft_consensus.cc:2781] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:09.911218 19578 raft_consensus.cc:686] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 LEADER]: Becoming Leader. State: Replica: 71df639daaff42e2a8cc0e38e2a92350, State: Running, Role: LEADER
I20260925 14:52:09.911644 19578 consensus_queue.cc:237] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } }
I20260925 14:52:09.915720 19065 catalog_manager.cc:5495] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71df639daaff42e2a8cc0e38e2a92350 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "71df639daaff42e2a8cc0e38e2a92350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:09.981869 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:09.975756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.17.133.193:43043)
0925 14:52:09.975972 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:09.975978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:09.976111 (+ 133us) server_negotiation.cc:407] Connection header received
0925 14:52:09.976445 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:09.976450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:09.976565 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:09.976780 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:09.977121 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.977915 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.979272 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:09.979692 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:09.979832 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:09.980298 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:09.980877 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:09.981025 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:09.981142 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 14:52:09.981219 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:52:09.989519 19362 raft_consensus.cc:1261] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Refusing update from remote peer 71df639daaff42e2a8cc0e38e2a92350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:09.989521 19497 raft_consensus.cc:1261] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Refusing update from remote peer 71df639daaff42e2a8cc0e38e2a92350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:09.990576 19581 consensus_queue.cc:1035] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:09.990928 19578 consensus_queue.cc:1035] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:10.002079 19584 mvcc.cc:205] Tried to move back new op lower bound from 7333265121227939840 to 7333265120921362432. Current Snapshot: MvccSnapshot[applied={T|T < 7333265121227939840}]
I20260925 14:52:10.022377 19065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:10.022658 19065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:10.026330 19065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714: Sending DeleteTablet for 3 replicas of tablet 927ef5d5838c4c5aaaf45b1572925f35
I20260925 14:52:10.027369 19203 tablet_service.cc:1514] Processing DeleteTablet for tablet 927ef5d5838c4c5aaaf45b1572925f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:10 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:42682
I20260925 14:52:10.027385 19342 tablet_service.cc:1514] Processing DeleteTablet for tablet 927ef5d5838c4c5aaaf45b1572925f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:10 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:42222
I20260925 14:52:10.028031 19481 tablet_service.cc:1514] Processing DeleteTablet for tablet 927ef5d5838c4c5aaaf45b1572925f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:10 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:49574
I20260925 14:52:10.029184 19065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:10.029825 19065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:10.039348 19341 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7fe6f3840f498eab616fcd17a03017 (DEFAULT_TABLE table=test-table [id=59ba7bb74780459a8dbcf28bf16f65c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:10.040009 19341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7fe6f3840f498eab616fcd17a03017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:10.041064 19595 tablet_replica.cc:330] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: stopping tablet replica
I20260925 14:52:10.041289 19480 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7fe6f3840f498eab616fcd17a03017 (DEFAULT_TABLE table=test-table [id=59ba7bb74780459a8dbcf28bf16f65c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:10.041632 19595 raft_consensus.cc:2229] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:10.041867 19480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7fe6f3840f498eab616fcd17a03017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:10.042109 19595 raft_consensus.cc:2258] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:10.044093 19595 ts_tablet_manager.cc:1905] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:10.053876 19202 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7fe6f3840f498eab616fcd17a03017 (DEFAULT_TABLE table=test-table [id=59ba7bb74780459a8dbcf28bf16f65c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:10.054630 19202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7fe6f3840f498eab616fcd17a03017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:10.056771 19595 ts_tablet_manager.cc:1918] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:10.056944 19595 log.cc:1196] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/927ef5d5838c4c5aaaf45b1572925f35
I20260925 14:52:10.057595 19595 ts_tablet_manager.cc:1939] T 927ef5d5838c4c5aaaf45b1572925f35 P 3cba4d073bad4b61a5801de5fbe2a32c: Deleting consensus metadata
I20260925 14:52:10.058177 19594 tablet_replica.cc:330] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: stopping tablet replica
I20260925 14:52:10.058758 19596 tablet_replica.cc:330] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: stopping tablet replica
I20260925 14:52:10.059407 19052 catalog_manager.cc:4841] TS 3cba4d073bad4b61a5801de5fbe2a32c (127.17.133.194:34615): tablet 927ef5d5838c4c5aaaf45b1572925f35 (table test-table [id=8c94e139a3eb42aba4a5454f1e198f5a]) successfully deleted
I20260925 14:52:10.059950 19594 raft_consensus.cc:2229] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:10.060467 19596 raft_consensus.cc:2229] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:10.063261 19570 tablet_bootstrap.cc:492] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: Bootstrap starting.
I20260925 14:52:10.064244 19572 tablet_bootstrap.cc:492] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: Bootstrap starting.
I20260925 14:52:10.064888 19570 tablet_bootstrap.cc:654] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:10.065804 19572 tablet_bootstrap.cc:654] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:10.065083 19571 tablet_bootstrap.cc:492] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: Bootstrap starting.
I20260925 14:52:10.068291 19572 tablet_bootstrap.cc:492] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: No bootstrap required, opened a new log
I20260925 14:52:10.068450 19572 ts_tablet_manager.cc:1397] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:52:10.069274 19596 raft_consensus.cc:2258] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:10.069696 19571 tablet_bootstrap.cc:654] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:10.069272 19572 raft_consensus.cc:348] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.069891 19572 raft_consensus.cc:374] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:10.069989 19572 raft_consensus.cc:729] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cba4d073bad4b61a5801de5fbe2a32c, State: Initialized, Role: FOLLOWER
I20260925 14:52:10.070135 19594 raft_consensus.cc:2258] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:10.070305 19572 consensus_queue.cc:260] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.073743 19572 ts_tablet_manager.cc:1428] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260925 14:52:10.074119 19594 ts_tablet_manager.cc:1905] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:10.078984 19571 tablet_bootstrap.cc:492] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: No bootstrap required, opened a new log
I20260925 14:52:10.079172 19571 ts_tablet_manager.cc:1397] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260925 14:52:10.080013 19570 tablet_bootstrap.cc:492] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: No bootstrap required, opened a new log
I20260925 14:52:10.080049 19571 raft_consensus.cc:348] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.080214 19570 ts_tablet_manager.cc:1397] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20260925 14:52:10.080276 19571 raft_consensus.cc:374] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:10.080374 19571 raft_consensus.cc:729] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df639daaff42e2a8cc0e38e2a92350, State: Initialized, Role: FOLLOWER
I20260925 14:52:10.080694 19571 consensus_queue.cc:260] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.080991 19594 ts_tablet_manager.cc:1918] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:10.081138 19594 log.cc:1196] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/927ef5d5838c4c5aaaf45b1572925f35
I20260925 14:52:10.081261 19571 ts_tablet_manager.cc:1428] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:10.081328 19570 raft_consensus.cc:348] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.081636 19594 ts_tablet_manager.cc:1939] T 927ef5d5838c4c5aaaf45b1572925f35 P 71df639daaff42e2a8cc0e38e2a92350: Deleting consensus metadata
I20260925 14:52:10.081640 19570 raft_consensus.cc:374] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:10.081837 19570 raft_consensus.cc:729] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df5f86546eb4c539c5ad079a369bbff, State: Initialized, Role: FOLLOWER
I20260925 14:52:10.082173 19570 consensus_queue.cc:260] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.082751 19596 ts_tablet_manager.cc:1905] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:10.083245 19053 catalog_manager.cc:4841] TS 71df639daaff42e2a8cc0e38e2a92350 (127.17.133.193:43043): tablet 927ef5d5838c4c5aaaf45b1572925f35 (table test-table [id=8c94e139a3eb42aba4a5454f1e198f5a]) successfully deleted
I20260925 14:52:10.085438 19570 ts_tablet_manager.cc:1428] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 14:52:10.093888 19596 ts_tablet_manager.cc:1918] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:10.094053 19596 log.cc:1196] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/927ef5d5838c4c5aaaf45b1572925f35
I20260925 14:52:10.094631 19596 ts_tablet_manager.cc:1939] T 927ef5d5838c4c5aaaf45b1572925f35 P 5df5f86546eb4c539c5ad079a369bbff: Deleting consensus metadata
I20260925 14:52:10.096771 19051 catalog_manager.cc:4841] TS 5df5f86546eb4c539c5ad079a369bbff (127.17.133.195:44889): tablet 927ef5d5838c4c5aaaf45b1572925f35 (table test-table [id=8c94e139a3eb42aba4a5454f1e198f5a]) successfully deleted
I20260925 14:52:10.160192 19590 raft_consensus.cc:482] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:10.160465 19590 raft_consensus.cc:504] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.161245 19590 leader_election.cc:288] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cba4d073bad4b61a5801de5fbe2a32c (127.17.133.194:34615), 5df5f86546eb4c539c5ad079a369bbff (127.17.133.195:44889)
I20260925 14:52:10.161741 19362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7fe6f3840f498eab616fcd17a03017" candidate_uuid: "71df639daaff42e2a8cc0e38e2a92350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" is_pre_election: true
I20260925 14:52:10.161808 19497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7fe6f3840f498eab616fcd17a03017" candidate_uuid: "71df639daaff42e2a8cc0e38e2a92350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df5f86546eb4c539c5ad079a369bbff" is_pre_election: true
I20260925 14:52:10.162000 19362 raft_consensus.cc:2454] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df639daaff42e2a8cc0e38e2a92350 in term 0.
I20260925 14:52:10.162123 19497 raft_consensus.cc:2454] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df639daaff42e2a8cc0e38e2a92350 in term 0.
I20260925 14:52:10.162519 19155 leader_election.cc:302] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cba4d073bad4b61a5801de5fbe2a32c, 71df639daaff42e2a8cc0e38e2a92350; no voters:
I20260925 14:52:10.162845 19590 raft_consensus.cc:2781] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:10.162971 19590 raft_consensus.cc:482] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:10.163079 19590 raft_consensus.cc:3037] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:10.165392 19590 raft_consensus.cc:504] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.165974 19590 leader_election.cc:288] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [CANDIDATE]: Term 1 election: Requested vote from peers 3cba4d073bad4b61a5801de5fbe2a32c (127.17.133.194:34615), 5df5f86546eb4c539c5ad079a369bbff (127.17.133.195:44889)
I20260925 14:52:10.166342 19362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7fe6f3840f498eab616fcd17a03017" candidate_uuid: "71df639daaff42e2a8cc0e38e2a92350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cba4d073bad4b61a5801de5fbe2a32c"
I20260925 14:52:10.166415 19497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7fe6f3840f498eab616fcd17a03017" candidate_uuid: "71df639daaff42e2a8cc0e38e2a92350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df5f86546eb4c539c5ad079a369bbff"
I20260925 14:52:10.166524 19362 raft_consensus.cc:3037] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:10.166610 19497 raft_consensus.cc:3037] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:10.168475 19497 raft_consensus.cc:2454] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df639daaff42e2a8cc0e38e2a92350 in term 1.
I20260925 14:52:10.168975 19154 leader_election.cc:302] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5df5f86546eb4c539c5ad079a369bbff, 71df639daaff42e2a8cc0e38e2a92350; no voters:
I20260925 14:52:10.169258 19590 raft_consensus.cc:2781] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:10.169421 19590 raft_consensus.cc:686] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 LEADER]: Becoming Leader. State: Replica: 71df639daaff42e2a8cc0e38e2a92350, State: Running, Role: LEADER
I20260925 14:52:10.169697 19590 consensus_queue.cc:237] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.173755 19065 catalog_manager.cc:5495] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71df639daaff42e2a8cc0e38e2a92350 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "71df639daaff42e2a8cc0e38e2a92350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:10.174188 19362 raft_consensus.cc:2454] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df639daaff42e2a8cc0e38e2a92350 in term 1.
Sep 25 14:52:10 dist-test-slave-v5vl krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:10.245997 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.230918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41161 (local address 127.0.0.1:55716)
0925 14:52:10.231979 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:10.232004 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:10.232149 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:10.232618 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:10.232622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:10.233508 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:10.233782 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.233793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.234901 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.234907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.236435 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.236451 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.237196 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.237202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.237380 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.238067 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:10.238090 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:10.240572 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:10.243497 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:10.243510 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:10.243518 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:10.243906 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:10.244244 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:10.244249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:10.244252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:10.244390 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:10.244949 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:10.244957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:10.245116 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 14:52:10.245283 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 14:52:10.245559 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":720,"threads_started":1}
I20260925 14:52:10.246016 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.230940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.17.133.254:41161)
0925 14:52:10.232308 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.232312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.232330 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:52:10.232373 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.232376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.232426 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.232515 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:10.233956 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.234744 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.236563 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.237041 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.237132 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.240805 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:10.240809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:10.240813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:10.240837 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:10.243362 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.244017 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.244026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.244029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.244136 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.244498 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.244502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.244505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.244823 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:10.244961 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.245356 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.245667 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":1270,"thread_start_us":1145,"threads_started":1}
I20260925 14:52:10.250869 19065 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:55716:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:52:10.251847 19065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:10.258311 19341 tablet_service.cc:1467] Processing CreateTablet for tablet 5491cd65b2304aff89fe3da367c00eaa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=325f4717a1d04f6f94aad212bf9492ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:52:10.258276 19202 tablet_service.cc:1467] Processing CreateTablet for tablet 5491cd65b2304aff89fe3da367c00eaa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=325f4717a1d04f6f94aad212bf9492ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:52:10.258786 19341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5491cd65b2304aff89fe3da367c00eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:10.258811 19202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5491cd65b2304aff89fe3da367c00eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:10.262631 19480 tablet_service.cc:1467] Processing CreateTablet for tablet 5491cd65b2304aff89fe3da367c00eaa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=325f4717a1d04f6f94aad212bf9492ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:52:10.263118 19480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5491cd65b2304aff89fe3da367c00eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:10.265362 19572 tablet_bootstrap.cc:492] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: Bootstrap starting.
I20260925 14:52:10.266479 19571 tablet_bootstrap.cc:492] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: Bootstrap starting.
I20260925 14:52:10.266906 19572 tablet_bootstrap.cc:654] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:10.268160 19571 tablet_bootstrap.cc:654] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:10.270262 19570 tablet_bootstrap.cc:492] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: Bootstrap starting.
I20260925 14:52:10.272137 19570 tablet_bootstrap.cc:654] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:10.278298 19572 tablet_bootstrap.cc:492] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: No bootstrap required, opened a new log
I20260925 14:52:10.278483 19572 ts_tablet_manager.cc:1397] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260925 14:52:10.279471 19572 raft_consensus.cc:348] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.279760 19572 raft_consensus.cc:374] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:10.279855 19572 raft_consensus.cc:729] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cba4d073bad4b61a5801de5fbe2a32c, State: Initialized, Role: FOLLOWER
I20260925 14:52:10.280130 19572 consensus_queue.cc:260] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.280603 19577 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.280956 19577 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:10.281090 19571 tablet_bootstrap.cc:492] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: No bootstrap required, opened a new log
I20260925 14:52:10.281250 19571 ts_tablet_manager.cc:1397] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260925 14:52:10.281997 19572 ts_tablet_manager.cc:1428] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 14:52:10.281607 19570 tablet_bootstrap.cc:492] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: No bootstrap required, opened a new log
I20260925 14:52:10.283054 19570 ts_tablet_manager.cc:1397] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260925 14:52:10.283022 19571 raft_consensus.cc:348] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.283249 19571 raft_consensus.cc:374] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:10.283340 19571 raft_consensus.cc:729] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df639daaff42e2a8cc0e38e2a92350, State: Initialized, Role: FOLLOWER
I20260925 14:52:10.283593 19571 consensus_queue.cc:260] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.284164 19571 ts_tablet_manager.cc:1428] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 14:52:10.284030 19590 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.284313 19590 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:10.285182 19570 raft_consensus.cc:348] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.285440 19570 raft_consensus.cc:374] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:10.285526 19570 raft_consensus.cc:729] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df5f86546eb4c539c5ad079a369bbff, State: Initialized, Role: FOLLOWER
I20260925 14:52:10.285758 19570 consensus_queue.cc:260] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.286281 19570 ts_tablet_manager.cc:1428] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:52:10.286211 19576 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.286521 19576 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:10.344867 19576 raft_consensus.cc:482] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:10.345124 19576 raft_consensus.cc:504] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.347441 19576 leader_election.cc:288] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71df639daaff42e2a8cc0e38e2a92350 (127.17.133.193:43043), 3cba4d073bad4b61a5801de5fbe2a32c (127.17.133.194:34615)
I20260925 14:52:10.354800 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.347663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:34615 (local address 127.17.133.195:50273)
0925 14:52:10.348627 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:10.348646 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:10.348885 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:10.349470 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:10.349475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:10.349504 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:10.349835 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.349849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.351346 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.351352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.352973 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.352984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.354036 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.354042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.354215 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.354238 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:52:10.354348 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:52:10.354475 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":641,"threads_started":1}
I20260925 14:52:10.354782 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.347975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:50273 (local address 127.17.133.194:34615)
0925 14:52:10.348154 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.348160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.348736 (+ 576us) server_negotiation.cc:407] Connection header received
0925 14:52:10.349074 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.349083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.349167 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.349237 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:10.350046 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.351186 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.353115 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.353865 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.354030 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.354105 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.354372 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.354493 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:52:10.355419 19362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5491cd65b2304aff89fe3da367c00eaa" candidate_uuid: "5df5f86546eb4c539c5ad079a369bbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" is_pre_election: true
I20260925 14:52:10.355765 19362 raft_consensus.cc:2454] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5df5f86546eb4c539c5ad079a369bbff in term 0.
I20260925 14:52:10.356321 19434 leader_election.cc:302] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cba4d073bad4b61a5801de5fbe2a32c, 5df5f86546eb4c539c5ad079a369bbff; no voters:
I20260925 14:52:10.356773 19576 raft_consensus.cc:2781] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:10.356901 19576 raft_consensus.cc:482] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:10.357003 19576 raft_consensus.cc:3037] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:10.357268 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.347733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:36353 (local address 127.17.133.193:43043)
0925 14:52:10.347978 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.347988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.349820 (+ 1832us) server_negotiation.cc:407] Connection header received
0925 14:52:10.350077 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.350083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.350158 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.350235 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:10.350861 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.351941 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.354791 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.356100 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.356321 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.356395 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.356804 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.356946 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:52:10.357398 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.348206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43043 (local address 127.17.133.195:36353)
0925 14:52:10.349718 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:10.349737 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:10.349917 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:10.350436 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:10.350439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:10.350456 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:10.350718 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.350724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.352086 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.352092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.354590 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.354606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.356333 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.356343 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.356527 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.356619 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 14:52:10.356775 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 14:52:10.356957 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"mutex_wait_us":1,"thread_start_us":1223,"threads_started":1}
I20260925 14:52:10.358217 19228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5491cd65b2304aff89fe3da367c00eaa" candidate_uuid: "5df5f86546eb4c539c5ad079a369bbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df639daaff42e2a8cc0e38e2a92350" is_pre_election: true
I20260925 14:52:10.358670 19228 raft_consensus.cc:2454] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5df5f86546eb4c539c5ad079a369bbff in term 0.
I20260925 14:52:10.360163 19576 raft_consensus.cc:504] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.361351 19576 leader_election.cc:288] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [CANDIDATE]: Term 1 election: Requested vote from peers 71df639daaff42e2a8cc0e38e2a92350 (127.17.133.193:43043), 3cba4d073bad4b61a5801de5fbe2a32c (127.17.133.194:34615)
I20260925 14:52:10.361754 19362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5491cd65b2304aff89fe3da367c00eaa" candidate_uuid: "5df5f86546eb4c539c5ad079a369bbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cba4d073bad4b61a5801de5fbe2a32c"
I20260925 14:52:10.361778 19228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5491cd65b2304aff89fe3da367c00eaa" candidate_uuid: "5df5f86546eb4c539c5ad079a369bbff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df639daaff42e2a8cc0e38e2a92350"
I20260925 14:52:10.362010 19362 raft_consensus.cc:3037] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:10.362020 19228 raft_consensus.cc:3037] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:10.364111 19362 raft_consensus.cc:2454] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5df5f86546eb4c539c5ad079a369bbff in term 1.
I20260925 14:52:10.364470 19434 leader_election.cc:302] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cba4d073bad4b61a5801de5fbe2a32c, 5df5f86546eb4c539c5ad079a369bbff; no voters:
I20260925 14:52:10.364777 19576 raft_consensus.cc:2781] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:10.365296 19576 raft_consensus.cc:686] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [term 1 LEADER]: Becoming Leader. State: Replica: 5df5f86546eb4c539c5ad079a369bbff, State: Running, Role: LEADER
I20260925 14:52:10.365428 19228 raft_consensus.cc:2454] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5df5f86546eb4c539c5ad079a369bbff in term 1.
I20260925 14:52:10.365724 19576 consensus_queue.cc:237] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } }
I20260925 14:52:10.367946 19613 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: TxnStatusTablet state changed. Reason: New leader 5df5f86546eb4c539c5ad079a369bbff. Latest consensus state: current_term: 1 leader_uuid: "5df5f86546eb4c539c5ad079a369bbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.368305 19613 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:52:10.370028 19065 catalog_manager.cc:5495] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff reported cstate change: term changed from 0 to 1, leader changed from <none> to 5df5f86546eb4c539c5ad079a369bbff (127.17.133.195). New cstate: current_term: 1 leader_uuid: "5df5f86546eb4c539c5ad079a369bbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } health_report { overall_health: HEALTHY } } }
I20260925 14:52:10.370716 19615 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:52:10.370875 19615 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:52:10.408030 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.401014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:44889 (local address 127.0.0.1:49588)
0925 14:52:10.401215 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:10.401237 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:10.401414 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:10.402793 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:10.402798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:10.402809 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:52:10.402997 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.403007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.404243 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.404248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.405254 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.405272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.406312 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.406317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.406493 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.406540 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:52:10.407358 (+ 818us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:52:10.407377 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 14:52:10.407465 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:52:10.407603 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:52:10.408993 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.401119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.17.133.195:44889)
0925 14:52:10.402411 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.402416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.402431 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:52:10.402502 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.402505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.402577 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.402783 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:10.403162 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.404071 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.405409 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.406129 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.406282 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.406753 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:10.407250 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:10.407409 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.407486 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.407564 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1197,"thread_start_us":1100,"threads_started":1}
I20260925 14:52:10.412515 19362 raft_consensus.cc:1261] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Refusing update from remote peer 5df5f86546eb4c539c5ad079a369bbff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:10.412946 19228 raft_consensus.cc:1261] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350 [term 1 FOLLOWER]: Refusing update from remote peer 5df5f86546eb4c539c5ad079a369bbff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:10.413347 19576 consensus_queue.cc:1035] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:10.413656 19613 consensus_queue.cc:1035] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:10.425683 19577 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: TxnStatusTablet state changed. Reason: New leader 5df5f86546eb4c539c5ad079a369bbff. Latest consensus state: current_term: 1 leader_uuid: "5df5f86546eb4c539c5ad079a369bbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.426000 19577 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:10.428115 19613 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5df5f86546eb4c539c5ad079a369bbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.428376 19613 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:52:10.429890 19586 mvcc.cc:205] Tried to move back new op lower bound from 7333265122963017728 to 7333265122781765632. Current Snapshot: MvccSnapshot[applied={T|T < 7333265122963017728 or (T in {7333265122963017728})}]
I20260925 14:52:10.431785 19577 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5df5f86546eb4c539c5ad079a369bbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.432005 19577 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 3cba4d073bad4b61a5801de5fbe2a32c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:10.437642 19590 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: TxnStatusTablet state changed. Reason: New leader 5df5f86546eb4c539c5ad079a369bbff. Latest consensus state: current_term: 1 leader_uuid: "5df5f86546eb4c539c5ad079a369bbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.437971 19590 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:10.441195 19576 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5df5f86546eb4c539c5ad079a369bbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.441479 19576 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 5df5f86546eb4c539c5ad079a369bbff: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:52:10.445828 19590 tablet_replica.cc:439] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5df5f86546eb4c539c5ad079a369bbff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df639daaff42e2a8cc0e38e2a92350" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43043 } } peers { permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 } } peers { permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 } } }
I20260925 14:52:10.446127 19590 tablet_replica.cc:442] T 5491cd65b2304aff89fe3da367c00eaa P 71df639daaff42e2a8cc0e38e2a92350: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:10.468055 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.455948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:38987 (local address 127.17.133.195:44889)
0925 14:52:10.456117 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.456121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.461251 (+ 5130us) server_negotiation.cc:407] Connection header received
0925 14:52:10.461475 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.461480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.461576 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.461677 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:10.462454 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.463638 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.465941 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.466618 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.466739 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.467294 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:10.467431 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:10.467529 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.467688 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.467777 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:52:10.468571 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.455803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:44889 (local address 127.17.133.193:38987)
0925 14:52:10.461120 (+ 5317us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:10.461141 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:10.461321 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:10.461718 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:10.461723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:10.461735 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:52:10.462204 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.462218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.463824 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.463830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.465793 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.465809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.466816 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.466828 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.467015 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.467089 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:52:10.467542 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:52:10.467556 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:52:10.467649 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:52:10.467793 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5123,"thread_start_us":4980,"threads_started":1}
I20260925 14:52:10.481269 19362 raft_consensus.cc:1261] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Refusing update from remote peer 71df639daaff42e2a8cc0e38e2a92350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:10.481287 19497 raft_consensus.cc:1261] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Refusing update from remote peer 71df639daaff42e2a8cc0e38e2a92350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:10.482491 19590 consensus_queue.cc:1035] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cba4d073bad4b61a5801de5fbe2a32c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:10.482771 19578 consensus_queue.cc:1035] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df5f86546eb4c539c5ad079a369bbff" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:10.498450 19585 mvcc.cc:205] Tried to move back new op lower bound from 7333265123243593728 to 7333265121978036224. Current Snapshot: MvccSnapshot[applied={T|T < 7333265123243593728}]
I20260925 14:52:10.584649 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.536216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:41319 (local address 127.17.133.193:43043)
0925 14:52:10.536413 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.536419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.574230 (+ 37811us) server_negotiation.cc:407] Connection header received
0925 14:52:10.574591 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.574597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.574675 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.574729 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:10.578501 (+ 3772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.579761 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.582446 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.583111 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.583197 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.583875 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:10.584013 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:10.584082 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.584269 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.584347 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:52:10.585428 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.535851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43043 (local address 127.17.133.195:41319)
0925 14:52:10.574109 (+ 38258us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:10.574135 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:10.574422 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:10.574907 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:10.574911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:10.574923 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:52:10.575311 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.575321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.580129 (+ 4808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.580135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.582293 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:10.582307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.583384 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.583390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:10.583556 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.583635 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:52:10.584140 (+ 505us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:52:10.584149 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 14:52:10.584907 (+ 758us) client_negotiation.cc:240] Negotiation successful
0925 14:52:10.585043 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37759,"thread_start_us":37853,"threads_started":1}
I20260925 14:52:10.634264 19065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:10.634509 19065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:10.637434 19065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e91df70185c04666ab70422ede015714: Sending DeleteTablet for 3 replicas of tablet 2d7fe6f3840f498eab616fcd17a03017
I20260925 14:52:10.638259 19202 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7fe6f3840f498eab616fcd17a03017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:10 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:42682
I20260925 14:52:10.638310 19341 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7fe6f3840f498eab616fcd17a03017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:10 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:42222
I20260925 14:52:10.638751 19480 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7fe6f3840f498eab616fcd17a03017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:10 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:49574
I20260925 14:52:10.639020 19632 tablet_replica.cc:330] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: stopping tablet replica
I20260925 14:52:10.639272 19632 raft_consensus.cc:2229] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:10.639685 19632 raft_consensus.cc:2258] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:10.640928 19632 ts_tablet_manager.cc:1905] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:10.647789 19633 tablet_replica.cc:330] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: stopping tablet replica
I20260925 14:52:10.648104 19633 raft_consensus.cc:2229] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:10.648226 19634 tablet_replica.cc:330] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: stopping tablet replica
I20260925 14:52:10.648429 19634 raft_consensus.cc:2229] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:10.648494 19633 raft_consensus.cc:2258] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:10.648741 19634 raft_consensus.cc:2258] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:10.649932 19634 ts_tablet_manager.cc:1905] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:10.650898 19633 ts_tablet_manager.cc:1905] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:10.656805 19632 ts_tablet_manager.cc:1918] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:52:10.657064 19632 log.cc:1196] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/2d7fe6f3840f498eab616fcd17a03017
I20260925 14:52:10.657424 19634 ts_tablet_manager.cc:1918] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:52:10.657562 19634 log.cc:1196] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/2d7fe6f3840f498eab616fcd17a03017
I20260925 14:52:10.658042 19634 ts_tablet_manager.cc:1939] T 2d7fe6f3840f498eab616fcd17a03017 P 5df5f86546eb4c539c5ad079a369bbff: Deleting consensus metadata
I20260925 14:52:10.659034 19632 ts_tablet_manager.cc:1939] T 2d7fe6f3840f498eab616fcd17a03017 P 71df639daaff42e2a8cc0e38e2a92350: Deleting consensus metadata
I20260925 14:52:10.659255 19051 catalog_manager.cc:4841] TS 5df5f86546eb4c539c5ad079a369bbff (127.17.133.195:44889): tablet 2d7fe6f3840f498eab616fcd17a03017 (table test-table [id=59ba7bb74780459a8dbcf28bf16f65c5]) successfully deleted
I20260925 14:52:10.660040 19053 catalog_manager.cc:4841] TS 71df639daaff42e2a8cc0e38e2a92350 (127.17.133.193:43043): tablet 2d7fe6f3840f498eab616fcd17a03017 (table test-table [id=59ba7bb74780459a8dbcf28bf16f65c5]) successfully deleted
I20260925 14:52:10.662437 19633 ts_tablet_manager.cc:1918] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:52:10.662569 19633 log.cc:1196] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/2d7fe6f3840f498eab616fcd17a03017
I20260925 14:52:10.663029 19633 ts_tablet_manager.cc:1939] T 2d7fe6f3840f498eab616fcd17a03017 P 3cba4d073bad4b61a5801de5fbe2a32c: Deleting consensus metadata
I20260925 14:52:10.664129 19052 catalog_manager.cc:4841] TS 3cba4d073bad4b61a5801de5fbe2a32c (127.17.133.194:34615): tablet 2d7fe6f3840f498eab616fcd17a03017 (table test-table [id=59ba7bb74780459a8dbcf28bf16f65c5]) successfully deleted
Sep 25 14:52:10 dist-test-slave-v5vl krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:10.674513 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.648653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.17.133.193:43043)
0925 14:52:10.648822 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.648831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.657836 (+ 9005us) server_negotiation.cc:407] Connection header received
0925 14:52:10.658717 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.658723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.658820 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.658895 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:10.660866 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.661973 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.664074 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.664544 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.664776 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.668707 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:10.668720 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:10.668735 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:10.668773 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:10.671485 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.672190 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.672205 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.672208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.672369 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.672833 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.672839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.672843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.673335 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:10.673656 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.673919 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.674146 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 14:52:10.675247 19268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42694
I20260925 14:52:10.690937 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.678088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.17.133.254:41161)
0925 14:52:10.678365 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.678375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.678748 (+ 373us) server_negotiation.cc:407] Connection header received
0925 14:52:10.678936 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.678941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.679051 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.679141 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:10.680900 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.681652 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.682834 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.683280 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.683425 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.685801 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:10.685806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:10.685809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:10.685841 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:10.688340 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.689083 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.689090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.689094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.689217 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.689570 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.689574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.689577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.689847 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:10.690105 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.690413 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.690627 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 14:52:10.691556 19065 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55726
I20260925 14:52:10.693390 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19137
I20260925 14:52:10.707837 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.694729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.17.133.193:43043)
0925 14:52:10.695245 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.695251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.695269 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:52:10.695324 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.695329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.695403 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.695508 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:10.696544 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.697674 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.698792 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.699433 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.699568 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.702143 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:10.702150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:10.702155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:10.702198 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:10.705505 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.706070 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.706076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.706080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.706216 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.706510 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.706516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.706521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.706897 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:10.707132 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.707299 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.707500 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
W20260925 14:52:10.708436 19268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42696
W20260925 14:52:10.708972 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19137: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:52:10.709146 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 19137
I20260925 14:52:10.732774 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19277
Sep 25 14:52:10 dist-test-slave-v5vl krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:52:10.747556 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.733542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.17.133.194:34615)
0925 14:52:10.733756 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.733761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.733781 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:52:10.733936 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.733942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.734009 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.734149 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:10.735312 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.735993 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.737321 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.737724 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.737856 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.741975 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:10.741988 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:10.742002 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:10.742032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:10.745015 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.745649 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.745662 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.745665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.745794 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.746141 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.746148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.746151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.746467 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:10.746794 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.747005 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.747206 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260925 14:52:10.748363 19407 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42232
W20260925 14:52:10.749051 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19277: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:52:10.749225 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 19277
I20260925 14:52:10.771036 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19416
Sep 25 14:52:10 dist-test-slave-v5vl krb5kdc[19021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:52:10.787925 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:10.771808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.17.133.195:44889)
0925 14:52:10.772025 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:10.772033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:10.772056 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:52:10.772159 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:10.772165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:10.772269 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:10.772392 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:10.773696 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.774883 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.776042 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:10.776720 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:10.776860 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:10.781201 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:10.781219 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:10.781244 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:10.781287 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:10.785232 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.785825 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.785834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.785837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.785978 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:10.786308 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:10.786315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:10.786323 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:10.786840 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:10.787139 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:10.787318 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 14:52:10.787527 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 14:52:10.788887 19546 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49596
W20260925 14:52:10.789510 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19416: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:52:10.789652 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 19416
I20260925 14:52:10.809533 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19034
I20260925 14:52:10.810634 19106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:10.922034 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 19034
2026-09-25T14:52:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4628 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:11 dist-test-slave-v5vl krb5kdc[19643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:11 dist-test-slave-v5vl krb5kdc[19643](info): set up 2 sockets
Sep 25 14:52:11 dist-test-slave-v5vl krb5kdc[19643](info): commencing operation
krb5kdc: starting...
W20260925 14:52:12.055635 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 14:52:12.289388 17943 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.234s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 14:52:12.515633 17943 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.226s user 0.000s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 14:52:12.772907 17943 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.257s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/test-user.keytab.
W20260925 14:52:13.513983 17943 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.741s user 0.000s sys 0.004s
Sep 25 14:52:13 dist-test-slave-v5vl krb5kdc[19643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
W20260925 14:52:14.821801 17943 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.17.133.254: real 1.261s user 0.002s sys 0.006s
I20260925 14:52:14.822868 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45331
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:40159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:45331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:15.155467 19656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:15.155740 19656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:15.155826 19656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:15.162839 19656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:15.162951 19656 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:52:15.162992 19656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:15.163044 19656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:15.163092 19656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:15.163147 19656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:15.173127 19656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:40159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:45331
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:15.174371 19656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:15.175655 19656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:15.187271 19661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.187373 19664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.187476 19662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:15.188503 19656 server_base.cc:1034] running on GCE node
I20260925 14:52:15.189278 19656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:15.190675 19656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:15.191896 19656 hybrid_clock.cc:648] HybridClock initialized: now 1790347935191817 us; error 49 us; skew 500 ppm
I20260925 14:52:15.192194 19656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:15 dist-test-slave-v5vl krb5kdc[19643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:15.200939 19656 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:52:15.202669 19656 webserver.cc:460] Webserver started at http://127.17.133.254:34803/ using document root <none> and password file <none>
I20260925 14:52:15.203539 19656 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:15.203687 19656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:15.204035 19656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:15.207305 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "317e0a28e2c14ddcafdf9ef499517ad0"
format_stamp: "Formatted at 2026-09-25 14:52:15 on dist-test-slave-v5vl"
server_key: "8d5184835c2127600b6159c107f272c9"
server_key_iv: "76f55a816b9906ee80a09577b74d9377"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.208108 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "317e0a28e2c14ddcafdf9ef499517ad0"
format_stamp: "Formatted at 2026-09-25 14:52:15 on dist-test-slave-v5vl"
server_key: "8d5184835c2127600b6159c107f272c9"
server_key_iv: "76f55a816b9906ee80a09577b74d9377"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.213415 19656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 14:52:15.216976 19671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.218247 19656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:52:15.218442 19656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "317e0a28e2c14ddcafdf9ef499517ad0"
format_stamp: "Formatted at 2026-09-25 14:52:15 on dist-test-slave-v5vl"
server_key: "8d5184835c2127600b6159c107f272c9"
server_key_iv: "76f55a816b9906ee80a09577b74d9377"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.218652 19656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:15.231765 19656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:15.236208 19656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:15.236501 19656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:15.259832 19656 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:45331
I20260925 14:52:15.259858 19732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:45331 every 8 connection(s)
I20260925 14:52:15.261298 19656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:15.265167 19733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:15.269980 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 19656
I20260925 14:52:15.270197 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:15.270848 17943 external_mini_cluster.cc:1428] Setting key a77baea9760b0d4a214b73eb2dd858e3
I20260925 14:52:15.275965 19733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0: Bootstrap starting.
I20260925 14:52:15.279686 19733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:15.280768 19733 log.cc:824] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:15.284014 19733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0: No bootstrap required, opened a new log
Sep 25 14:52:15 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:15.291643 19733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317e0a28e2c14ddcafdf9ef499517ad0" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45331 } }
I20260925 14:52:15.292037 19733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:15.292148 19733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317e0a28e2c14ddcafdf9ef499517ad0, State: Initialized, Role: FOLLOWER
I20260925 14:52:15.292878 19733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317e0a28e2c14ddcafdf9ef499517ad0" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45331 } }
I20260925 14:52:15.293002 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.272844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.17.133.254:45331)
0925 14:52:15.273624 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.273636 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.273677 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:52:15.275287 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.275308 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.275707 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.276084 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:15.277495 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.278787 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.280090 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.280832 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.281012 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.285555 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:15.285562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:15.285576 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:15.285616 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:15.288747 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.289425 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.289431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.289437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.289611 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.290006 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.290010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.290012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.290536 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:15.290879 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.291317 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.291717 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
I20260925 14:52:15.293128 19733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:15.293270 19733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:15.293479 19733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:15.344357 19733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317e0a28e2c14ddcafdf9ef499517ad0" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45331 } }
I20260925 14:52:15.345038 19733 leader_election.cc:302] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 317e0a28e2c14ddcafdf9ef499517ad0; no voters:
I20260925 14:52:15.347031 19738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:15.347281 19733 leader_election.cc:288] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:15.347692 19738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [term 1 LEADER]: Becoming Leader. State: Replica: 317e0a28e2c14ddcafdf9ef499517ad0, State: Running, Role: LEADER
I20260925 14:52:15.348348 19738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317e0a28e2c14ddcafdf9ef499517ad0" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45331 } }
I20260925 14:52:15.349647 19733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:15.351163 19739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 317e0a28e2c14ddcafdf9ef499517ad0. Latest consensus state: current_term: 1 leader_uuid: "317e0a28e2c14ddcafdf9ef499517ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317e0a28e2c14ddcafdf9ef499517ad0" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45331 } } }
I20260925 14:52:15.351539 19739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:15.351089 19740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "317e0a28e2c14ddcafdf9ef499517ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317e0a28e2c14ddcafdf9ef499517ad0" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45331 } } }
I20260925 14:52:15.352170 19740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:15.354061 19744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:15.359611 19744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:15.367751 19744 catalog_manager.cc:1347] Generated new cluster ID: c9564e8697b14206a0e37620d77eef9e
I20260925 14:52:15.367866 19744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:15.394594 19744 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:15.396420 19744 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:15.405133 19744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0: Generated new TSK 0
I20260925 14:52:15.406138 19744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:52:15.467803 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45331
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:40159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:15.785373 19759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:15.785667 19759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:15.785856 19759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:15.793579 19759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:15.793751 19759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:15.793900 19759 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:52:15.793987 19759 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:52:15.794130 19759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:15.803646 19759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:40159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.133.254:45331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:15.805096 19759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:15.806475 19759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:15.818881 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.818845 19765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.818924 19764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:15.819438 19759 server_base.cc:1034] running on GCE node
I20260925 14:52:15.820055 19759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:15.820972 19759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:15.822158 19759 hybrid_clock.cc:648] HybridClock initialized: now 1790347935822126 us; error 28 us; skew 500 ppm
I20260925 14:52:15.822458 19759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:15 dist-test-slave-v5vl krb5kdc[19643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:15.825529 19759 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:52:15.826939 19759 webserver.cc:460] Webserver started at http://127.17.133.193:38767/ using document root <none> and password file <none>
I20260925 14:52:15.827620 19759 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:15.827754 19759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:15.828038 19759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:15.830915 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "eb41897f145a408b9d1289f9f5098bf7"
format_stamp: "Formatted at 2026-09-25 14:52:15 on dist-test-slave-v5vl"
server_key: "aa91907bfcba70f0ec10acb5deea5f88"
server_key_iv: "a71ffd3b9f167a473349410820d38f57"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.831663 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "eb41897f145a408b9d1289f9f5098bf7"
format_stamp: "Formatted at 2026-09-25 14:52:15 on dist-test-slave-v5vl"
server_key: "aa91907bfcba70f0ec10acb5deea5f88"
server_key_iv: "a71ffd3b9f167a473349410820d38f57"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.836557 19759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 14:52:15.839996 19774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.841264 19759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:15.841435 19759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "eb41897f145a408b9d1289f9f5098bf7"
format_stamp: "Formatted at 2026-09-25 14:52:15 on dist-test-slave-v5vl"
server_key: "aa91907bfcba70f0ec10acb5deea5f88"
server_key_iv: "a71ffd3b9f167a473349410820d38f57"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.841635 19759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:15.858695 19759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:15.862963 19759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:15.863220 19759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:15.865993 19759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:15.866142 19759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.866271 19759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:15.866367 19759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:52:15 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:15.889505 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.866325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:57743 (local address 127.17.133.254:45331)
0925 14:52:15.868959 (+ 2634us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.868963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.868980 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:52:15.871318 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.871326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.871406 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.871568 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:15.873684 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.875030 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.877252 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.877700 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.877832 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.882493 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:15.882500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:15.882506 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:15.882550 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:15.885834 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.886936 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.886944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.886950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.887126 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.887579 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.887586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.887592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.888036 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:15.888220 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.888939 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.889139 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2528,"thread_start_us":2443,"threads_started":1}
I20260925 14:52:15.891053 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.867008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45331 (local address 127.17.133.193:57743)
0925 14:52:15.868029 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.868079 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.870901 (+ 2822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.871868 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.871884 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.872643 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:15.873463 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.873490 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.875224 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.875231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.877105 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.877116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.877923 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.877933 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.878140 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.878863 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:15.878893 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:15.882214 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:15.885969 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.885979 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.886002 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.886765 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.887259 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.887262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.887264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.887458 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.888173 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:15.888178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:15.888495 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.888854 (+ 359us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.889357 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":434,"threads_started":1}
I20260925 14:52:15.912319 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.897379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45331 (local address 127.17.133.193:49355)
0925 14:52:15.897996 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.898011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.898113 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.898466 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.898469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.898882 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:15.899137 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.899145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.900877 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.900882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.902385 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.902396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.903382 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.903392 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.903635 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.904617 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:15.904641 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:15.905621 (+ 980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:15.909448 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.909456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.909463 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.910123 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.910535 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.910540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.910544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.910737 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.911454 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:15.911465 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:15.911648 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.911756 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.911984 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":333,"threads_started":1}
I20260925 14:52:15.912688 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.897366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:49355 (local address 127.17.133.254:45331)
0925 14:52:15.897586 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.897595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.898085 (+ 490us) server_negotiation.cc:407] Connection header received
0925 14:52:15.898226 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.898232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.898326 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.898456 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:15.899267 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.900716 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.902528 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.903146 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.903280 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.905813 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:15.905823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:15.905831 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:15.905876 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:15.909307 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.910283 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.910287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.910289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.910413 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.910876 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.910884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.910897 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.911297 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:15.911479 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.911901 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.912205 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:52:15.920374 19759 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:44853
I20260925 14:52:15.920454 19894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:44853 every 8 connection(s)
I20260925 14:52:15.921674 19759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:15.923659 19895 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45331
I20260925 14:52:15.923878 19895 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:15.924502 19895 heartbeater.cc:511] Master 127.17.133.254:45331 requested a full tablet report, sending...
I20260925 14:52:15.925640 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 19759
I20260925 14:52:15.925849 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:15.926476 17943 external_mini_cluster.cc:1428] Setting key 80bbba51d6905adac63a869ff4c075a2
I20260925 14:52:15.926469 19688 ts_manager.cc:194] Registered new tserver with Master: eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853)
I20260925 14:52:15.928059 19688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:57743
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:52:15.965340 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45331
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:40159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:16.276736 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:16.276994 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:16.277168 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:16.284317 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:16.284405 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:16.284504 19899 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:52:16.284606 19899 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:52:16.284721 19899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:16.294941 19899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:40159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.133.254:45331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:16.296329 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:16.297739 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:16.310662 19907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:16.310768 19905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:16.310691 19904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:16.311404 19899 server_base.cc:1034] running on GCE node
I20260925 14:52:16.311926 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:16.312788 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:16.313975 19899 hybrid_clock.cc:648] HybridClock initialized: now 1790347936313944 us; error 27 us; skew 500 ppm
I20260925 14:52:16.314270 19899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:16 dist-test-slave-v5vl krb5kdc[19643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347936, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:16.317235 19899 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:52:16.318570 19899 webserver.cc:460] Webserver started at http://127.17.133.194:36839/ using document root <none> and password file <none>
I20260925 14:52:16.319219 19899 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:16.319346 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:16.319680 19899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:16.322938 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "acda07d8333f4ce6a5ea13190b6db950"
format_stamp: "Formatted at 2026-09-25 14:52:16 on dist-test-slave-v5vl"
server_key: "c7b43d3ca160f908b93ff75b051196d8"
server_key_iv: "789bc467b5d57fa71b9183f0d596496c"
server_key_version: "encryptionkey@0"
I20260925 14:52:16.323930 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "acda07d8333f4ce6a5ea13190b6db950"
format_stamp: "Formatted at 2026-09-25 14:52:16 on dist-test-slave-v5vl"
server_key: "c7b43d3ca160f908b93ff75b051196d8"
server_key_iv: "789bc467b5d57fa71b9183f0d596496c"
server_key_version: "encryptionkey@0"
I20260925 14:52:16.329231 19899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:52:16.332463 19914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:16.333717 19899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:16.333889 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "acda07d8333f4ce6a5ea13190b6db950"
format_stamp: "Formatted at 2026-09-25 14:52:16 on dist-test-slave-v5vl"
server_key: "c7b43d3ca160f908b93ff75b051196d8"
server_key_iv: "789bc467b5d57fa71b9183f0d596496c"
server_key_version: "encryptionkey@0"
I20260925 14:52:16.334087 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:16.354822 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:16.359232 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:16.359540 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:16.362151 19899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:16.362294 19899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:16.362421 19899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:16.362521 19899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:52:16 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347936, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:16.387982 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.362924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:45285 (local address 127.17.133.254:45331)
0925 14:52:16.363109 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:16.363116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:16.364557 (+ 1441us) server_negotiation.cc:407] Connection header received
0925 14:52:16.367597 (+ 3040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:16.367602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:16.367734 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:16.367820 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:16.370590 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.371585 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.374880 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.375423 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.375574 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.380723 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:16.380732 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:16.380741 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:16.380788 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:16.384349 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.385467 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.385471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.385474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.385593 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.386176 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.386179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.386181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.386431 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:16.386540 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:16.387583 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0925 14:52:16.387744 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:52:16.390117 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.363559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45331 (local address 127.17.133.194:45285)
0925 14:52:16.364385 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:16.364438 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:16.367234 (+ 2796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:16.368142 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:16.368156 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:16.369172 (+ 1016us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:16.370269 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:16.370340 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.371765 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.371773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:16.374708 (+ 2935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:16.374727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.375729 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.375740 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:16.376119 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.377028 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:16.377053 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:16.380412 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:16.384494 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:16.384507 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:16.384527 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:16.385319 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:16.385705 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:16.385713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:16.385719 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:16.386039 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:16.386561 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:16.386572 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:16.387140 (+ 568us) client_negotiation.cc:769] Sending connection context
0925 14:52:16.387505 (+ 365us) client_negotiation.cc:240] Negotiation successful
0925 14:52:16.388191 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":302,"threads_started":1}
I20260925 14:52:16.412570 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.399293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45331 (local address 127.17.133.194:60523)
0925 14:52:16.399857 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:16.399879 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:16.400030 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:16.400344 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:16.400349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:16.400838 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:16.401291 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:16.401304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.402733 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.402739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:16.403737 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:16.403747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.404547 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.404554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:16.404741 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.405534 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:16.405551 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:16.406239 (+ 688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:16.409491 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:16.409499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:16.409505 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:16.410295 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:16.410698 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:16.410703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:16.410707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:16.410892 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:16.411706 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:16.411709 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:16.411850 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:52:16.411953 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:52:16.412190 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":320,"threads_started":1}
I20260925 14:52:16.412895 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.399125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:60523 (local address 127.17.133.254:45331)
0925 14:52:16.399282 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:16.399286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:16.399964 (+ 678us) server_negotiation.cc:407] Connection header received
0925 14:52:16.400146 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:16.400151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:16.400217 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:16.400324 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:16.401498 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.402581 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.403875 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.404311 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.404435 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.406418 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:16.406424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:16.406431 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:16.406465 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:16.409328 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.410420 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.410427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.410433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.410573 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.411031 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.411040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.411046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.411555 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:16.411750 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:16.412095 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 14:52:16.412375 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:52:16.420727 19899 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:45347
I20260925 14:52:16.420789 20033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:45347 every 8 connection(s)
I20260925 14:52:16.421909 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:16.422891 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 19899
I20260925 14:52:16.423079 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:16.423651 17943 external_mini_cluster.cc:1428] Setting key ed9e17168b4ad3229315dd712f3bbcf2
I20260925 14:52:16.423969 20034 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45331
I20260925 14:52:16.424571 20034 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:16.425267 20034 heartbeater.cc:511] Master 127.17.133.254:45331 requested a full tablet report, sending...
I20260925 14:52:16.426597 19688 ts_manager.cc:194] Registered new tserver with Master: acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194:45347)
I20260925 14:52:16.427850 19688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:45285
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:52:16.462028 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45331
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:40159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:16.773968 20038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:16.774221 20038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:16.774356 20038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:16.781450 20038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:16.781558 20038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:16.781720 20038 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:52:16.781809 20038 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:52:16.781951 20038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:16.791281 20038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:40159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.133.254:45331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:16.792572 20038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:16.793938 20038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:16.806557 20043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:16.806702 20044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:16.806591 20046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:16.807569 20038 server_base.cc:1034] running on GCE node
I20260925 14:52:16.808254 20038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:16.809190 20038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:16.810405 20038 hybrid_clock.cc:648] HybridClock initialized: now 1790347936810361 us; error 57 us; skew 500 ppm
I20260925 14:52:16.810712 20038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:16 dist-test-slave-v5vl krb5kdc[19643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347936, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:16.813696 20038 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:52:16.815095 20038 webserver.cc:460] Webserver started at http://127.17.133.195:40093/ using document root <none> and password file <none>
I20260925 14:52:16.815728 20038 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:16.815861 20038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:16.816156 20038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:16.818903 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "7496e357ecb34e188080263ab41816d6"
format_stamp: "Formatted at 2026-09-25 14:52:16 on dist-test-slave-v5vl"
server_key: "7b002e3ac8fb333411866d4c84ed5679"
server_key_iv: "bbbaf4dc8b1cce2b9b5ddfa88ade1f24"
server_key_version: "encryptionkey@0"
I20260925 14:52:16.819669 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "7496e357ecb34e188080263ab41816d6"
format_stamp: "Formatted at 2026-09-25 14:52:16 on dist-test-slave-v5vl"
server_key: "7b002e3ac8fb333411866d4c84ed5679"
server_key_iv: "bbbaf4dc8b1cce2b9b5ddfa88ade1f24"
server_key_version: "encryptionkey@0"
I20260925 14:52:16.824703 20038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 14:52:16.827881 20053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:16.829210 20038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:52:16.829375 20038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "7496e357ecb34e188080263ab41816d6"
format_stamp: "Formatted at 2026-09-25 14:52:16 on dist-test-slave-v5vl"
server_key: "7b002e3ac8fb333411866d4c84ed5679"
server_key_iv: "bbbaf4dc8b1cce2b9b5ddfa88ade1f24"
server_key_version: "encryptionkey@0"
I20260925 14:52:16.829560 20038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:16.854161 20038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:16.858496 20038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:16.858800 20038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:16.861465 20038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:16.861583 20038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:16.861732 20038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:16.861852 20038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:52:16 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347936, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:16.885599 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.861822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:43293 (local address 127.17.133.254:45331)
0925 14:52:16.862013 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:16.862021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:16.863635 (+ 1614us) server_negotiation.cc:407] Connection header received
0925 14:52:16.865861 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:16.865870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:16.865973 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:16.866132 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:16.868438 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.869271 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.872760 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.873210 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.873311 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.877969 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:16.877979 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:16.877987 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:16.878041 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:16.881995 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.882916 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.882924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.882930 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.883129 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.883587 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.883595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.883600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.884034 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:16.884228 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:16.884999 (+ 771us) server_negotiation.cc:299] Negotiation successful
0925 14:52:16.885233 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:52:16.886624 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.862485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45331 (local address 127.17.133.195:43293)
0925 14:52:16.863430 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:16.863493 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:16.865512 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:16.866419 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:16.866427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:16.867149 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:16.868274 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:16.868297 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.869478 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.869485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:16.872565 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:16.872610 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.873472 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.873482 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:16.873853 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.874783 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:16.874824 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:16.877687 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:16.882149 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:16.882165 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:16.882186 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:16.882766 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:16.883256 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:16.883261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:16.883267 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:16.883458 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:16.884168 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:16.884174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:16.884511 (+ 337us) client_negotiation.cc:769] Sending connection context
0925 14:52:16.884859 (+ 348us) client_negotiation.cc:240] Negotiation successful
0925 14:52:16.885435 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":368,"threads_started":1}
I20260925 14:52:16.907256 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.892165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:59527 (local address 127.17.133.254:45331)
0925 14:52:16.892347 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:16.892355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:16.893195 (+ 840us) server_negotiation.cc:407] Connection header received
0925 14:52:16.893470 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:16.893477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:16.893558 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:16.893656 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:16.894982 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.896025 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.898125 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.898654 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.898796 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.901324 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:16.901332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:16.901337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:16.901377 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:16.904399 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.905176 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.905185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.905190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.905354 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.905853 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.905858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.905863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.906167 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:16.906311 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:16.906715 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 14:52:16.906917 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:52:16.907359 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.892246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45331 (local address 127.17.133.195:59527)
0925 14:52:16.893062 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:16.893085 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:16.893322 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:16.893679 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:16.893684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:16.894249 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:16.894844 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:16.894855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.896235 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.896243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:16.897949 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:16.897959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.898885 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.898895 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:16.899174 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.899958 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:16.899989 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:16.901065 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:16.904538 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:16.904543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:16.904547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:16.905009 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:16.905489 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:16.905496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:16.905505 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:16.905674 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:16.906291 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:16.906295 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:16.906469 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 14:52:16.906602 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 14:52:16.906916 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":418,"threads_started":1}
I20260925 14:52:16.918226 20038 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:39283
I20260925 14:52:16.918344 20172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:39283 every 8 connection(s)
I20260925 14:52:16.919507 20038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:16.921744 20173 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45331
I20260925 14:52:16.924017 20173 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:16.924669 20173 heartbeater.cc:511] Master 127.17.133.254:45331 requested a full tablet report, sending...
I20260925 14:52:16.925879 19688 ts_manager.cc:194] Registered new tserver with Master: 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283)
I20260925 14:52:16.926652 19688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:43293
I20260925 14:52:16.928807 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 20038
I20260925 14:52:16.928987 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:16.929396 17943 external_mini_cluster.cc:1428] Setting key 512a0410e2d1191e3bac4766aec77c53
I20260925 14:52:16.931331 19895 heartbeater.cc:503] Master 127.17.133.254:45331 was elected leader, sending a full tablet report...
I20260925 14:52:16.931983 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:52:16 dist-test-slave-v5vl krb5kdc[19643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:52:16 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:16.977084 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.963587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.17.133.254:45331)
0925 14:52:16.963846 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:16.963853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:16.964157 (+ 304us) server_negotiation.cc:407] Connection header received
0925 14:52:16.964369 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:16.964373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:16.964431 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:16.964536 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:16.965898 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.966637 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.967826 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.968236 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.968366 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.972083 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:16.972088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:16.972092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:16.972133 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:16.974508 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.975153 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.975160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.975163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.975293 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.975669 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.975677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.975680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.975946 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:16.976217 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:16.976483 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:52:16.976736 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 14:52:16.980919 19688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:16.987617 19688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:17.010490 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.000015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.17.133.194:45347)
0925 14:52:17.000547 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.000556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.001275 (+ 719us) server_negotiation.cc:407] Connection header received
0925 14:52:17.001770 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.001782 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.002114 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.002303 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:17.003489 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.004969 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.007805 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.009370 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.009580 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.009785 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.010013 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.010194 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
I20260925 14:52:17.010564 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.999694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:45347 (local address 127.0.0.1:57200)
0925 14:52:17.001123 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.001171 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.001456 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.002672 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.002684 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.002724 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:17.003269 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.003286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.005139 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.005150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.007631 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.007644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.009568 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.009575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.009774 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.009825 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.009957 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.010162 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":5,"thread_start_us":1029,"threads_started":1}
I20260925 14:52:17.011986 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.000059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:39283 (local address 127.0.0.1:37794)
0925 14:52:17.002551 (+ 2492us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.002574 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.002715 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.004681 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.004685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.004720 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:17.005237 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.005255 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.006886 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.006893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.009133 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.009143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.011063 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.011069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.011300 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.011334 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.011423 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.011563 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2325,"mutex_wait_us":5,"thread_start_us":2187,"threads_started":1}
I20260925 14:52:17.012025 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.001215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:44853 (local address 127.0.0.1:59664)
0925 14:52:17.003329 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.003348 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.003445 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.004401 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.004406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.004430 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:17.004883 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.004900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.006800 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.006806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.009439 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.009454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.011248 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.011253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.011449 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.011476 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.011549 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.011679 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1970,"mutex_wait_us":82,"thread_start_us":1782,"threads_started":1}
I20260925 14:52:17.012038 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.001895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.17.133.193:44853)
0925 14:52:17.003658 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.003666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.003693 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:52:17.003781 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.003789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.004027 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.004218 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:17.005140 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.006648 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.009616 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.011055 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.011241 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.011385 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.011608 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.011765 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":1480,"threads_started":1}
I20260925 14:52:17.012737 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.000702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.17.133.195:39283)
0925 14:52:17.004022 (+ 3320us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.004029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.004105 (+ 76us) server_negotiation.cc:407] Connection header received
0925 14:52:17.004194 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.004201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.004443 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.005434 (+ 991us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:17.005679 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.006743 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.009282 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.010715 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.010907 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.011086 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.012213 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.012376 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3170,"thread_start_us":3004,"threads_started":1}
I20260925 14:52:17.014967 19964 tablet_service.cc:1467] Processing CreateTablet for tablet 71adce09c1ee4a61b357c24ef0df3886 (DEFAULT_TABLE table=test-table [id=c1712c2851fd4c15ba8fa1138b5f5e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:17.015408 19825 tablet_service.cc:1467] Processing CreateTablet for tablet 71adce09c1ee4a61b357c24ef0df3886 (DEFAULT_TABLE table=test-table [id=c1712c2851fd4c15ba8fa1138b5f5e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:17.016448 19825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71adce09c1ee4a61b357c24ef0df3886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:17.016484 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71adce09c1ee4a61b357c24ef0df3886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:17.016078 20103 tablet_service.cc:1467] Processing CreateTablet for tablet 71adce09c1ee4a61b357c24ef0df3886 (DEFAULT_TABLE table=test-table [id=c1712c2851fd4c15ba8fa1138b5f5e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:17.017333 20103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71adce09c1ee4a61b357c24ef0df3886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:17.029076 20192 tablet_bootstrap.cc:492] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: Bootstrap starting.
I20260925 14:52:17.030925 20193 tablet_bootstrap.cc:492] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: Bootstrap starting.
I20260925 14:52:17.032841 20194 tablet_bootstrap.cc:492] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: Bootstrap starting.
I20260925 14:52:17.033605 20192 tablet_bootstrap.cc:654] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:17.034462 20193 tablet_bootstrap.cc:654] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:17.034869 20192 log.cc:824] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:17.035661 20193 log.cc:824] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:17.036418 20194 tablet_bootstrap.cc:654] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:17.037755 20192 tablet_bootstrap.cc:492] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: No bootstrap required, opened a new log
I20260925 14:52:17.037853 20194 log.cc:824] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:17.038039 20192 ts_tablet_manager.cc:1397] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 14:52:17.040874 20194 tablet_bootstrap.cc:492] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: No bootstrap required, opened a new log
I20260925 14:52:17.040975 20193 tablet_bootstrap.cc:492] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: No bootstrap required, opened a new log
I20260925 14:52:17.041189 20194 ts_tablet_manager.cc:1397] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 14:52:17.041285 20193 ts_tablet_manager.cc:1397] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260925 14:52:17.048285 20193 raft_consensus.cc:348] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } }
I20260925 14:52:17.048694 20193 raft_consensus.cc:374] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:17.048806 20193 raft_consensus.cc:729] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acda07d8333f4ce6a5ea13190b6db950, State: Initialized, Role: FOLLOWER
I20260925 14:52:17.048707 20192 raft_consensus.cc:348] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } }
I20260925 14:52:17.049110 20192 raft_consensus.cc:374] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:17.049216 20192 raft_consensus.cc:729] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb41897f145a408b9d1289f9f5098bf7, State: Initialized, Role: FOLLOWER
I20260925 14:52:17.049537 20193 consensus_queue.cc:260] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } }
I20260925 14:52:17.049830 20192 consensus_queue.cc:260] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } }
I20260925 14:52:17.051312 20193 ts_tablet_manager.cc:1428] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 14:52:17.051818 20034 heartbeater.cc:503] Master 127.17.133.254:45331 was elected leader, sending a full tablet report...
I20260925 14:52:17.052312 20194 raft_consensus.cc:348] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } }
I20260925 14:52:17.052860 20194 raft_consensus.cc:374] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:17.052975 20194 raft_consensus.cc:729] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7496e357ecb34e188080263ab41816d6, State: Initialized, Role: FOLLOWER
I20260925 14:52:17.053645 20194 consensus_queue.cc:260] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } }
I20260925 14:52:17.054456 20192 ts_tablet_manager.cc:1428] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: Time spent starting tablet: real 0.016s user 0.014s sys 0.004s
I20260925 14:52:17.055445 20173 heartbeater.cc:503] Master 127.17.133.254:45331 was elected leader, sending a full tablet report...
I20260925 14:52:17.056103 20194 ts_tablet_manager.cc:1428] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260925 14:52:17.171208 20174 tablet.cc:2369] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:17.173794 20035 tablet.cc:2369] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:17.174126 19896 tablet.cc:2369] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:17.230129 20198 raft_consensus.cc:482] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:17.230443 20198 raft_consensus.cc:504] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } }
I20260925 14:52:17.232026 20198 leader_election.cc:288] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283), eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853)
I20260925 14:52:17.239684 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.232461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:37851 (local address 127.17.133.193:44853)
0925 14:52:17.232700 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.232706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.233401 (+ 695us) server_negotiation.cc:407] Connection header received
0925 14:52:17.233643 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.233649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.233734 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.233828 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:17.234795 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.235833 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.237984 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.238796 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.238993 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.239068 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.239352 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.239462 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:52:17.239791 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.232565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:55873 (local address 127.17.133.195:39283)
0925 14:52:17.232805 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.232813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.233453 (+ 640us) server_negotiation.cc:407] Connection header received
0925 14:52:17.233643 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.233649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.233734 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.233828 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:17.234768 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.235835 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.238033 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.238918 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.239110 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.239180 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.239442 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.239550 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 14:52:17.239846 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.232309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:44853 (local address 127.17.133.194:37851)
0925 14:52:17.233296 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.233315 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.233477 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.233861 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.233866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.233893 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:17.234668 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.234682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.235980 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.235985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.237830 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.237841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.239020 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.239026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.239195 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.239217 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.239330 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.239440 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":148,"thread_start_us":736,"threads_started":1}
I20260925 14:52:17.239933 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.232420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:39283 (local address 127.17.133.194:55873)
0925 14:52:17.233353 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.233369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.233507 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.233861 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.233866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.233904 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:17.234547 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.234563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.236012 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.236015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.237895 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.237907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.239138 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.239144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.239294 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.239328 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.239420 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.239531 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":74,"spinlock_wait_cycles":46208,"thread_start_us":490,"threads_started":1}
I20260925 14:52:17.240904 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71adce09c1ee4a61b357c24ef0df3886" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb41897f145a408b9d1289f9f5098bf7" is_pre_election: true
I20260925 14:52:17.240935 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71adce09c1ee4a61b357c24ef0df3886" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6" is_pre_election: true
I20260925 14:52:17.241385 19845 raft_consensus.cc:2454] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 0.
I20260925 14:52:17.241427 20128 raft_consensus.cc:2454] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 0.
I20260925 14:52:17.242041 19915 leader_election.cc:302] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7496e357ecb34e188080263ab41816d6, acda07d8333f4ce6a5ea13190b6db950, eb41897f145a408b9d1289f9f5098bf7; no voters:
I20260925 14:52:17.242483 20198 raft_consensus.cc:2781] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:17.242626 20198 raft_consensus.cc:482] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:17.242733 20198 raft_consensus.cc:3037] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:17.244863 20198 raft_consensus.cc:504] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } }
I20260925 14:52:17.245437 20198 leader_election.cc:288] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 election: Requested vote from peers 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283), eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853)
I20260925 14:52:17.245883 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71adce09c1ee4a61b357c24ef0df3886" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6"
I20260925 14:52:17.245895 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71adce09c1ee4a61b357c24ef0df3886" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb41897f145a408b9d1289f9f5098bf7"
I20260925 14:52:17.246114 20128 raft_consensus.cc:3037] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:17.246122 19845 raft_consensus.cc:3037] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:17.249234 19845 raft_consensus.cc:2454] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 1.
I20260925 14:52:17.249234 20128 raft_consensus.cc:2454] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 1.
I20260925 14:52:17.249629 19917 leader_election.cc:302] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7496e357ecb34e188080263ab41816d6, acda07d8333f4ce6a5ea13190b6db950; no voters:
I20260925 14:52:17.249893 20198 raft_consensus.cc:2781] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:17.250340 20198 raft_consensus.cc:686] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 1 LEADER]: Becoming Leader. State: Replica: acda07d8333f4ce6a5ea13190b6db950, State: Running, Role: LEADER
I20260925 14:52:17.250871 20198 consensus_queue.cc:237] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } }
I20260925 14:52:17.255255 19686 catalog_manager.cc:5495] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 reported cstate change: term changed from 0 to 1, leader changed from <none> to acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194). New cstate: current_term: 1 leader_uuid: "acda07d8333f4ce6a5ea13190b6db950" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } health_report { overall_health: UNKNOWN } } }
Sep 25 14:52:17 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:17.332448 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.319138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45331 (local address 127.0.0.1:45566)
0925 14:52:17.319696 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.319711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.319856 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.320167 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.320177 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.320878 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:17.321113 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.321121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.322084 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.322091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.323064 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.323075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.323762 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.323766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.323869 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.324533 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:17.324549 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:17.327385 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:17.330371 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:17.330379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:17.330384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:17.330747 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:17.331045 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:17.331048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:17.331051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:17.331188 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:17.331647 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:17.331652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:17.331788 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.331933 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.332113 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":302,"threads_started":1}
I20260925 14:52:17.332455 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.319138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.17.133.254:45331)
0925 14:52:17.319304 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.319308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.319795 (+ 487us) server_negotiation.cc:407] Connection header received
0925 14:52:17.319969 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.319972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.320038 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.320128 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:17.321245 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.321967 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.323194 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.323611 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.323702 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.327579 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:17.327585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:17.327590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:17.327623 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:17.330253 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:17.330841 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:17.330844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:17.330846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:17.330956 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:17.331273 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:17.331276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:17.331278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:17.331549 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:17.331651 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.332015 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.332206 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":60}
I20260925 14:52:17.337235 19686 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:45566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:52:17.338402 19686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:17.344973 19964 tablet_service.cc:1467] Processing CreateTablet for tablet 142bf9e56167415784e6369ade94bd66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80c0049fe4714f93884a810ab21fe53a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:52:17.344813 19825 tablet_service.cc:1467] Processing CreateTablet for tablet 142bf9e56167415784e6369ade94bd66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80c0049fe4714f93884a810ab21fe53a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:52:17.345191 20103 tablet_service.cc:1467] Processing CreateTablet for tablet 142bf9e56167415784e6369ade94bd66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80c0049fe4714f93884a810ab21fe53a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:52:17.345798 19825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142bf9e56167415784e6369ade94bd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:17.345799 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142bf9e56167415784e6369ade94bd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:17.346058 20103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142bf9e56167415784e6369ade94bd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:17.351804 20194 tablet_bootstrap.cc:492] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: Bootstrap starting.
I20260925 14:52:17.352895 20193 tablet_bootstrap.cc:492] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: Bootstrap starting.
I20260925 14:52:17.352934 20192 tablet_bootstrap.cc:492] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: Bootstrap starting.
I20260925 14:52:17.353361 20194 tablet_bootstrap.cc:654] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:17.354377 20193 tablet_bootstrap.cc:654] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:17.354565 20192 tablet_bootstrap.cc:654] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:17.355753 20194 tablet_bootstrap.cc:492] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: No bootstrap required, opened a new log
I20260925 14:52:17.355907 20194 ts_tablet_manager.cc:1397] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 14:52:17.356458 20193 tablet_bootstrap.cc:492] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: No bootstrap required, opened a new log
I20260925 14:52:17.356638 20193 ts_tablet_manager.cc:1397] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260925 14:52:17.356966 20194 raft_consensus.cc:348] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.357308 20194 raft_consensus.cc:374] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:17.357245 20193 raft_consensus.cc:348] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.357434 20194 raft_consensus.cc:729] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7496e357ecb34e188080263ab41816d6, State: Initialized, Role: FOLLOWER
I20260925 14:52:17.357512 20193 raft_consensus.cc:374] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:17.357610 20193 raft_consensus.cc:729] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acda07d8333f4ce6a5ea13190b6db950, State: Initialized, Role: FOLLOWER
I20260925 14:52:17.357759 20194 consensus_queue.cc:260] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.357896 20193 consensus_queue.cc:260] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.358388 20193 ts_tablet_manager.cc:1428] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:17.358421 20194 ts_tablet_manager.cc:1428] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 14:52:17.358273 20200 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.358633 20200 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:17.358928 20198 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.359220 20198 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:17.365788 20192 tablet_bootstrap.cc:492] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: No bootstrap required, opened a new log
I20260925 14:52:17.365908 20192 ts_tablet_manager.cc:1397] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260925 14:52:17.366461 20192 raft_consensus.cc:348] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.366629 20192 raft_consensus.cc:374] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:17.366699 20192 raft_consensus.cc:729] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb41897f145a408b9d1289f9f5098bf7, State: Initialized, Role: FOLLOWER
I20260925 14:52:17.366912 20192 consensus_queue.cc:260] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.367519 20192 ts_tablet_manager.cc:1428] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 14:52:17.367380 20199 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.367769 20199 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:17.568750 20199 raft_consensus.cc:482] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:17.568995 20199 raft_consensus.cc:504] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.570499 20199 leader_election.cc:288] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194:45347), 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283)
I20260925 14:52:17.579212 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.570883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:34117 (local address 127.17.133.194:45347)
0925 14:52:17.572080 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.572088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.572106 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:52:17.572178 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.572182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.572247 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.572357 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:17.573112 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.574165 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.576955 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.578101 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.578235 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.578294 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.578794 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.578928 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":1010,"threads_started":1}
I20260925 14:52:17.579419 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.570872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:45347 (local address 127.17.133.193:34117)
0925 14:52:17.571557 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.571584 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.571743 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.572403 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.572408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.572504 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:17.572960 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.572968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.574307 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.574310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.576821 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.576833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.578313 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.578318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.578544 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.578661 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.578755 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.578961 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":347,"spinlock_wait_cycles":22272,"thread_start_us":759,"threads_started":1}
I20260925 14:52:17.579456 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.570891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:39283 (local address 127.17.133.193:46305)
0925 14:52:17.571990 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.572015 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.572114 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.572403 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.572408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.572431 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:17.572903 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.572921 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.574305 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.574309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.576825 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.576833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.578297 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.578305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.578539 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.578669 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.578755 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.578961 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":234,"thread_start_us":363,"threads_started":1}
I20260925 14:52:17.579970 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.570880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:46305 (local address 127.17.133.195:39283)
0925 14:52:17.571101 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.571109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.572112 (+ 1003us) server_negotiation.cc:407] Connection header received
0925 14:52:17.572206 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.572210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.572280 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.572368 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:17.573091 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.574166 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.576955 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.578035 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.578212 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.578284 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.578794 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.578928 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:52:17.580338 19984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142bf9e56167415784e6369ade94bd66" candidate_uuid: "eb41897f145a408b9d1289f9f5098bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acda07d8333f4ce6a5ea13190b6db950" is_pre_election: true
I20260925 14:52:17.580533 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142bf9e56167415784e6369ade94bd66" candidate_uuid: "eb41897f145a408b9d1289f9f5098bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6" is_pre_election: true
I20260925 14:52:17.580796 20128 raft_consensus.cc:2454] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb41897f145a408b9d1289f9f5098bf7 in term 0.
I20260925 14:52:17.580788 19984 raft_consensus.cc:2454] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb41897f145a408b9d1289f9f5098bf7 in term 0.
I20260925 14:52:17.581248 19777 leader_election.cc:302] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7496e357ecb34e188080263ab41816d6, eb41897f145a408b9d1289f9f5098bf7; no voters:
I20260925 14:52:17.581624 20199 raft_consensus.cc:2781] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:17.581753 20199 raft_consensus.cc:482] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:17.581858 20199 raft_consensus.cc:3037] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:17.583817 20199 raft_consensus.cc:504] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.584359 20199 leader_election.cc:288] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [CANDIDATE]: Term 1 election: Requested vote from peers acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194:45347), 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283)
I20260925 14:52:17.584790 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142bf9e56167415784e6369ade94bd66" candidate_uuid: "eb41897f145a408b9d1289f9f5098bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6"
I20260925 14:52:17.584808 19984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142bf9e56167415784e6369ade94bd66" candidate_uuid: "eb41897f145a408b9d1289f9f5098bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acda07d8333f4ce6a5ea13190b6db950"
I20260925 14:52:17.584978 19984 raft_consensus.cc:3037] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:17.584995 20128 raft_consensus.cc:3037] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:17.587790 20128 raft_consensus.cc:2454] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb41897f145a408b9d1289f9f5098bf7 in term 1.
I20260925 14:52:17.587805 19984 raft_consensus.cc:2454] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb41897f145a408b9d1289f9f5098bf7 in term 1.
I20260925 14:52:17.588169 19777 leader_election.cc:302] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7496e357ecb34e188080263ab41816d6, acda07d8333f4ce6a5ea13190b6db950, eb41897f145a408b9d1289f9f5098bf7; no voters:
I20260925 14:52:17.588431 20199 raft_consensus.cc:2781] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:17.588835 20199 raft_consensus.cc:686] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [term 1 LEADER]: Becoming Leader. State: Replica: eb41897f145a408b9d1289f9f5098bf7, State: Running, Role: LEADER
I20260925 14:52:17.589329 20199 consensus_queue.cc:237] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.592321 20217 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: TxnStatusTablet state changed. Reason: New leader eb41897f145a408b9d1289f9f5098bf7. Latest consensus state: current_term: 1 leader_uuid: "eb41897f145a408b9d1289f9f5098bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.592763 20217 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:52:17.593415 20219 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:52:17.593689 20219 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:52:17.593657 19686 catalog_manager.cc:5495] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "eb41897f145a408b9d1289f9f5098bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:17.632647 20198 consensus_queue.cc:1035] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:17.640568 20224 consensus_queue.cc:1035] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:17.654663 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.648256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.17.133.193:44853)
0925 14:52:17.648534 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.648540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.648951 (+ 411us) server_negotiation.cc:407] Connection header received
0925 14:52:17.649038 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.649043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.649125 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.649253 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:17.649758 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.650614 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.652545 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.652968 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.653065 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.653579 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:17.654125 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:17.654244 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.654365 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.654425 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260925 14:52:17.654863 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.648326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:44853 (local address 127.0.0.1:33244)
0925 14:52:17.648599 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.648620 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.648773 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.649421 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.649427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.649439 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:52:17.649617 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.649626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.650734 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.650741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.652415 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.652429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.653119 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.653126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.653330 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.653398 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:52:17.654231 (+ 833us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:52:17.654252 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.654342 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.654495 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 14:52:17.661677 20128 raft_consensus.cc:1261] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Refusing update from remote peer eb41897f145a408b9d1289f9f5098bf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:17.661808 19984 raft_consensus.cc:1261] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Refusing update from remote peer eb41897f145a408b9d1289f9f5098bf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:17.662379 20199 consensus_queue.cc:1035] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:17.662748 20217 consensus_queue.cc:1035] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:17.677088 20224 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: TxnStatusTablet state changed. Reason: New leader eb41897f145a408b9d1289f9f5098bf7. Latest consensus state: current_term: 1 leader_uuid: "eb41897f145a408b9d1289f9f5098bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.677425 20224 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:17.677420 20200 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: TxnStatusTablet state changed. Reason: New leader eb41897f145a408b9d1289f9f5098bf7. Latest consensus state: current_term: 1 leader_uuid: "eb41897f145a408b9d1289f9f5098bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.677670 20200 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:17.680569 20226 mvcc.cc:205] Tried to move back new op lower bound from 7333265152653774848 to 7333265152369098752. Current Snapshot: MvccSnapshot[applied={T|T < 7333265152653774848}]
I20260925 14:52:17.681324 20217 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb41897f145a408b9d1289f9f5098bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.681555 20217 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:52:17.688177 20199 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb41897f145a408b9d1289f9f5098bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.688450 20199 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P eb41897f145a408b9d1289f9f5098bf7: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:52:17.700312 20200 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb41897f145a408b9d1289f9f5098bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.700654 20200 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P 7496e357ecb34e188080263ab41816d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:17.702495 20224 tablet_replica.cc:439] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb41897f145a408b9d1289f9f5098bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } } }
I20260925 14:52:17.702757 20224 tablet_replica.cc:442] T 142bf9e56167415784e6369ade94bd66 P acda07d8333f4ce6a5ea13190b6db950: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:52:17.711836 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.704299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.17.133.194:45347)
0925 14:52:17.704491 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.704500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.706144 (+ 1644us) server_negotiation.cc:407] Connection header received
0925 14:52:17.706327 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.706333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.706430 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.706582 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:17.706958 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.708104 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.709487 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.710087 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.710212 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.710566 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:17.711232 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:17.711375 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.711420 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.711496 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:52:17.727504 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.722072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:43945 (local address 127.17.133.193:44853)
0925 14:52:17.722281 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.722288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.722611 (+ 323us) server_negotiation.cc:407] Connection header received
0925 14:52:17.722823 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.722827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.722957 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.723084 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:17.723539 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.724356 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.725738 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.726197 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.726320 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.726764 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:17.726859 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:17.726949 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.727102 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.727179 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:52:17.727589 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.721730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:44853 (local address 127.17.133.194:43945)
0925 14:52:17.722511 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.722525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.722682 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.723099 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.723104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.723116 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:52:17.723399 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.723406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.724495 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.724501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.725602 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.725612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.726376 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.726385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.726520 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.726573 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:52:17.726964 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:52:17.726983 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.727078 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.727216 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":492,"threads_started":1}
I20260925 14:52:17.764575 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.758122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:36655 (local address 127.17.133.194:45347)
0925 14:52:17.758459 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:17.758467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:17.758943 (+ 476us) server_negotiation.cc:407] Connection header received
0925 14:52:17.759126 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:17.759132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:17.759210 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:17.759301 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:17.759833 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.760995 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.762766 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.763310 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.763448 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.763831 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:17.763970 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:17.764060 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:17.764208 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:52:17.764277 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260925 14:52:17.764717 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:17.758121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:45347 (local address 127.17.133.193:36655)
0925 14:52:17.758836 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:17.758855 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:17.758999 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:17.759317 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:17.759324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:17.759338 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:52:17.759693 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.759701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.761101 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.761105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.762627 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:17.762638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:17.763490 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:17.763499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:17.763619 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:17.763666 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:52:17.764070 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:52:17.764082 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 14:52:17.764177 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 14:52:17.764306 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":354,"threads_started":1}
I20260925 14:52:17.801152 19686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:17.801434 19686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:17.805540 19686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0: Sending DeleteTablet for 3 replicas of tablet 71adce09c1ee4a61b357c24ef0df3886
I20260925 14:52:17.806645 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet 71adce09c1ee4a61b357c24ef0df3886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:17 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:57200
I20260925 14:52:17.806732 20103 tablet_service.cc:1514] Processing DeleteTablet for tablet 71adce09c1ee4a61b357c24ef0df3886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:17 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:37794
I20260925 14:52:17.808387 19825 tablet_service.cc:1514] Processing DeleteTablet for tablet 71adce09c1ee4a61b357c24ef0df3886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:17 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:59664
I20260925 14:52:17.816272 20245 tablet_replica.cc:330] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: stopping tablet replica
I20260925 14:52:17.816757 19686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:17.817551 19686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:17.817552 20245 raft_consensus.cc:2229] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:17.818342 20245 raft_consensus.cc:2258] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:17.825521 20244 tablet_replica.cc:330] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: stopping tablet replica
I20260925 14:52:17.826174 19963 tablet_service.cc:1467] Processing CreateTablet for tablet ef1ddc1d4fde4ebcba5b652d00bb7c29 (DEFAULT_TABLE table=test-table [id=6217e5baedec48c6ab8554f0f6ff48fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:17.826089 19825 tablet_service.cc:1467] Processing CreateTablet for tablet ef1ddc1d4fde4ebcba5b652d00bb7c29 (DEFAULT_TABLE table=test-table [id=6217e5baedec48c6ab8554f0f6ff48fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:17.826690 19963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1ddc1d4fde4ebcba5b652d00bb7c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:17.826957 19825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1ddc1d4fde4ebcba5b652d00bb7c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:17.834016 20192 tablet_bootstrap.cc:492] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: Bootstrap starting.
I20260925 14:52:17.835132 20193 tablet_bootstrap.cc:492] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: Bootstrap starting.
I20260925 14:52:17.835711 20192 tablet_bootstrap.cc:654] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:17.836426 20245 ts_tablet_manager.cc:1905] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:17.837013 20193 tablet_bootstrap.cc:654] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:17.825847 20102 tablet_service.cc:1467] Processing CreateTablet for tablet ef1ddc1d4fde4ebcba5b652d00bb7c29 (DEFAULT_TABLE table=test-table [id=6217e5baedec48c6ab8554f0f6ff48fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:17.846359 20102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1ddc1d4fde4ebcba5b652d00bb7c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:17.847061 20193 tablet_bootstrap.cc:492] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: No bootstrap required, opened a new log
I20260925 14:52:17.847226 20193 ts_tablet_manager.cc:1397] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260925 14:52:17.848012 20193 raft_consensus.cc:348] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.848250 20193 raft_consensus.cc:374] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:17.849284 20193 raft_consensus.cc:729] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acda07d8333f4ce6a5ea13190b6db950, State: Initialized, Role: FOLLOWER
I20260925 14:52:17.849529 20193 consensus_queue.cc:260] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.850277 20193 ts_tablet_manager.cc:1428] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 14:52:17.850805 20244 raft_consensus.cc:2229] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:17.851202 20244 raft_consensus.cc:2258] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:17.851889 20243 tablet_replica.cc:330] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: stopping tablet replica
I20260925 14:52:17.853729 20194 tablet_bootstrap.cc:492] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: Bootstrap starting.
I20260925 14:52:17.852566 20243 raft_consensus.cc:2229] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:17.855234 20244 ts_tablet_manager.cc:1905] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:17.855644 20243 raft_consensus.cc:2258] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:17.855696 20194 tablet_bootstrap.cc:654] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:17.855691 20245 ts_tablet_manager.cc:1918] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:52:17.855842 20245 log.cc:1196] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/71adce09c1ee4a61b357c24ef0df3886
I20260925 14:52:17.856735 20245 ts_tablet_manager.cc:1939] T 71adce09c1ee4a61b357c24ef0df3886 P eb41897f145a408b9d1289f9f5098bf7: Deleting consensus metadata
I20260925 14:52:17.858420 20194 tablet_bootstrap.cc:492] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: No bootstrap required, opened a new log
I20260925 14:52:17.858677 20194 ts_tablet_manager.cc:1397] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260925 14:52:17.859346 19672 catalog_manager.cc:4841] TS eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853): tablet 71adce09c1ee4a61b357c24ef0df3886 (table test-table [id=c1712c2851fd4c15ba8fa1138b5f5e73]) successfully deleted
I20260925 14:52:17.859654 20194 raft_consensus.cc:348] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.859931 20194 raft_consensus.cc:374] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:17.860023 20194 raft_consensus.cc:729] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7496e357ecb34e188080263ab41816d6, State: Initialized, Role: FOLLOWER
I20260925 14:52:17.860320 20194 consensus_queue.cc:260] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.861068 20194 ts_tablet_manager.cc:1428] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 14:52:17.861621 20192 tablet_bootstrap.cc:492] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: No bootstrap required, opened a new log
I20260925 14:52:17.861781 20192 ts_tablet_manager.cc:1397] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20260925 14:52:17.862509 20192 raft_consensus.cc:348] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.863009 20243 ts_tablet_manager.cc:1905] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:17.863332 20192 raft_consensus.cc:374] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:17.863471 20192 raft_consensus.cc:729] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb41897f145a408b9d1289f9f5098bf7, State: Initialized, Role: FOLLOWER
I20260925 14:52:17.863786 20192 consensus_queue.cc:260] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:17.865229 20192 ts_tablet_manager.cc:1428] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 14:52:17.871681 20243 ts_tablet_manager.cc:1918] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:52:17.871860 20243 log.cc:1196] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/71adce09c1ee4a61b357c24ef0df3886
I20260925 14:52:17.872051 20244 ts_tablet_manager.cc:1918] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:52:17.872257 20244 log.cc:1196] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/71adce09c1ee4a61b357c24ef0df3886
I20260925 14:52:17.872439 20243 ts_tablet_manager.cc:1939] T 71adce09c1ee4a61b357c24ef0df3886 P acda07d8333f4ce6a5ea13190b6db950: Deleting consensus metadata
I20260925 14:52:17.872857 20244 ts_tablet_manager.cc:1939] T 71adce09c1ee4a61b357c24ef0df3886 P 7496e357ecb34e188080263ab41816d6: Deleting consensus metadata
I20260925 14:52:17.874728 19674 catalog_manager.cc:4841] TS 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283): tablet 71adce09c1ee4a61b357c24ef0df3886 (table test-table [id=c1712c2851fd4c15ba8fa1138b5f5e73]) successfully deleted
I20260925 14:52:17.875016 19675 catalog_manager.cc:4841] TS acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194:45347): tablet 71adce09c1ee4a61b357c24ef0df3886 (table test-table [id=c1712c2851fd4c15ba8fa1138b5f5e73]) successfully deleted
I20260925 14:52:18.057737 20203 raft_consensus.cc:482] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:18.057977 20203 raft_consensus.cc:504] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:18.058804 20203 leader_election.cc:288] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853), 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283)
I20260925 14:52:18.059381 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1ddc1d4fde4ebcba5b652d00bb7c29" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb41897f145a408b9d1289f9f5098bf7" is_pre_election: true
I20260925 14:52:18.059515 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1ddc1d4fde4ebcba5b652d00bb7c29" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6" is_pre_election: true
I20260925 14:52:18.059652 19845 raft_consensus.cc:2454] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 0.
I20260925 14:52:18.059767 20128 raft_consensus.cc:2454] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 0.
I20260925 14:52:18.060278 19917 leader_election.cc:302] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7496e357ecb34e188080263ab41816d6, acda07d8333f4ce6a5ea13190b6db950; no voters:
I20260925 14:52:18.060657 20203 raft_consensus.cc:2781] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:18.060792 20203 raft_consensus.cc:482] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:18.060883 20203 raft_consensus.cc:3037] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:18.064215 20203 raft_consensus.cc:504] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:18.064895 20203 leader_election.cc:288] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 election: Requested vote from peers eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853), 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283)
I20260925 14:52:18.065290 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1ddc1d4fde4ebcba5b652d00bb7c29" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb41897f145a408b9d1289f9f5098bf7"
I20260925 14:52:18.065562 19845 raft_consensus.cc:3037] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:18.065738 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1ddc1d4fde4ebcba5b652d00bb7c29" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6"
I20260925 14:52:18.065971 20128 raft_consensus.cc:3037] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:18.068535 19845 raft_consensus.cc:2454] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 1.
I20260925 14:52:18.068933 20128 raft_consensus.cc:2454] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 1.
I20260925 14:52:18.069074 19915 leader_election.cc:302] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acda07d8333f4ce6a5ea13190b6db950, eb41897f145a408b9d1289f9f5098bf7; no voters:
I20260925 14:52:18.069494 20203 raft_consensus.cc:2781] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:18.069725 20203 raft_consensus.cc:686] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 1 LEADER]: Becoming Leader. State: Replica: acda07d8333f4ce6a5ea13190b6db950, State: Running, Role: LEADER
I20260925 14:52:18.070060 20203 consensus_queue.cc:237] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:18.072932 19687 catalog_manager.cc:5495] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 reported cstate change: term changed from 0 to 1, leader changed from <none> to acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194). New cstate: current_term: 1 leader_uuid: "acda07d8333f4ce6a5ea13190b6db950" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:18.098516 20128 raft_consensus.cc:1261] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Refusing update from remote peer acda07d8333f4ce6a5ea13190b6db950: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:18.099018 19845 raft_consensus.cc:1261] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Refusing update from remote peer acda07d8333f4ce6a5ea13190b6db950: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:18.098992 20224 consensus_queue.cc:1035] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:18.099778 20224 consensus_queue.cc:1035] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:52:18.122563 20222 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:52:19.702852 19686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:19.703094 19686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:19.707263 19686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0: Sending DeleteTablet for 3 replicas of tablet ef1ddc1d4fde4ebcba5b652d00bb7c29
I20260925 14:52:19.707954 19825 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1ddc1d4fde4ebcba5b652d00bb7c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:19 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:59664
I20260925 14:52:19.708062 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1ddc1d4fde4ebcba5b652d00bb7c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:19 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:57200
I20260925 14:52:19.708101 20102 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1ddc1d4fde4ebcba5b652d00bb7c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:19 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:37794
I20260925 14:52:19.709766 20273 tablet_replica.cc:330] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: stopping tablet replica
I20260925 14:52:19.709923 20274 tablet_replica.cc:330] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: stopping tablet replica
I20260925 14:52:19.710019 20273 raft_consensus.cc:2229] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:19.710297 20273 raft_consensus.cc:2258] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:19.710367 20275 tablet_replica.cc:330] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: stopping tablet replica
I20260925 14:52:19.710532 20275 raft_consensus.cc:2229] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:19.710659 20274 raft_consensus.cc:2229] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:19.710813 20275 raft_consensus.cc:2258] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:19.711064 20274 raft_consensus.cc:2258] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:19.711736 20275 ts_tablet_manager.cc:1905] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:19.711939 20273 ts_tablet_manager.cc:1905] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:19.712077 20274 ts_tablet_manager.cc:1905] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:19.717768 20274 ts_tablet_manager.cc:1918] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:52:19.717916 20274 log.cc:1196] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/ef1ddc1d4fde4ebcba5b652d00bb7c29
I20260925 14:52:19.718245 20273 ts_tablet_manager.cc:1918] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:52:19.718286 20275 ts_tablet_manager.cc:1918] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:52:19.718369 20274 ts_tablet_manager.cc:1939] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P acda07d8333f4ce6a5ea13190b6db950: Deleting consensus metadata
I20260925 14:52:19.718420 20273 log.cc:1196] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/ef1ddc1d4fde4ebcba5b652d00bb7c29
I20260925 14:52:19.718454 20275 log.cc:1196] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/ef1ddc1d4fde4ebcba5b652d00bb7c29
I20260925 14:52:19.718926 20275 ts_tablet_manager.cc:1939] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P 7496e357ecb34e188080263ab41816d6: Deleting consensus metadata
I20260925 14:52:19.718931 20273 ts_tablet_manager.cc:1939] T ef1ddc1d4fde4ebcba5b652d00bb7c29 P eb41897f145a408b9d1289f9f5098bf7: Deleting consensus metadata
I20260925 14:52:19.719483 19675 catalog_manager.cc:4841] TS acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194:45347): tablet ef1ddc1d4fde4ebcba5b652d00bb7c29 (table test-table [id=6217e5baedec48c6ab8554f0f6ff48fe]) successfully deleted
I20260925 14:52:19.719926 19674 catalog_manager.cc:4841] TS 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283): tablet ef1ddc1d4fde4ebcba5b652d00bb7c29 (table test-table [id=6217e5baedec48c6ab8554f0f6ff48fe]) successfully deleted
I20260925 14:52:19.720157 19672 catalog_manager.cc:4841] TS eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853): tablet ef1ddc1d4fde4ebcba5b652d00bb7c29 (table test-table [id=6217e5baedec48c6ab8554f0f6ff48fe]) successfully deleted
Sep 25 14:52:19 dist-test-slave-v5vl krb5kdc[19643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:52:19 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:19.757841 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:19.744146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.17.133.254:45331)
0925 14:52:19.744778 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:19.744782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:19.744804 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:52:19.744975 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:19.744979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:19.745038 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:19.745127 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:19.746426 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:19.747201 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:19.748334 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:19.748771 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:19.748868 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:19.752848 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:19.752854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:19.752857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:19.752889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:19.755480 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:19.756067 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:19.756071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:19.756073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:19.756181 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:19.756636 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:19.756639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:19.756641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:19.756907 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:19.757091 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:19.757387 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 14:52:19.757548 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":346,"threads_started":1}
I20260925 14:52:19.761286 19686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:19.761879 19686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:19.768203 19964 tablet_service.cc:1467] Processing CreateTablet for tablet c432410f6e78454ab6651b594ce8cbbe (DEFAULT_TABLE table=test-table [id=d30d1cf518914dbb841ca2a270058c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:19.768297 19825 tablet_service.cc:1467] Processing CreateTablet for tablet c432410f6e78454ab6651b594ce8cbbe (DEFAULT_TABLE table=test-table [id=d30d1cf518914dbb841ca2a270058c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:19.768646 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c432410f6e78454ab6651b594ce8cbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:19.768775 19825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c432410f6e78454ab6651b594ce8cbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:19.768703 20102 tablet_service.cc:1467] Processing CreateTablet for tablet c432410f6e78454ab6651b594ce8cbbe (DEFAULT_TABLE table=test-table [id=d30d1cf518914dbb841ca2a270058c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:19.769091 20102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c432410f6e78454ab6651b594ce8cbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:19.773614 20284 tablet_bootstrap.cc:492] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: Bootstrap starting.
I20260925 14:52:19.774927 20284 tablet_bootstrap.cc:654] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:19.775530 20285 tablet_bootstrap.cc:492] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: Bootstrap starting.
I20260925 14:52:19.777761 20285 tablet_bootstrap.cc:654] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:19.777944 20286 tablet_bootstrap.cc:492] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: Bootstrap starting.
I20260925 14:52:19.778558 20284 tablet_bootstrap.cc:492] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: No bootstrap required, opened a new log
I20260925 14:52:19.778690 20284 ts_tablet_manager.cc:1397] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 14:52:19.779469 20284 raft_consensus.cc:348] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:19.779701 20286 tablet_bootstrap.cc:654] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:19.779738 20284 raft_consensus.cc:374] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:19.779848 20284 raft_consensus.cc:729] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb41897f145a408b9d1289f9f5098bf7, State: Initialized, Role: FOLLOWER
I20260925 14:52:19.780085 20284 consensus_queue.cc:260] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:19.780433 20285 tablet_bootstrap.cc:492] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: No bootstrap required, opened a new log
I20260925 14:52:19.780568 20285 ts_tablet_manager.cc:1397] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 14:52:19.780644 20284 ts_tablet_manager.cc:1428] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:19.781381 20285 raft_consensus.cc:348] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:19.781626 20285 raft_consensus.cc:374] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:19.781713 20286 tablet_bootstrap.cc:492] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: No bootstrap required, opened a new log
I20260925 14:52:19.781714 20285 raft_consensus.cc:729] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acda07d8333f4ce6a5ea13190b6db950, State: Initialized, Role: FOLLOWER
I20260925 14:52:19.781899 20286 ts_tablet_manager.cc:1397] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260925 14:52:19.781916 20285 consensus_queue.cc:260] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:19.782626 20285 ts_tablet_manager.cc:1428] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:52:19.782636 20286 raft_consensus.cc:348] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:19.783236 20286 raft_consensus.cc:374] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:19.783366 20286 raft_consensus.cc:729] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7496e357ecb34e188080263ab41816d6, State: Initialized, Role: FOLLOWER
I20260925 14:52:19.783628 20286 consensus_queue.cc:260] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:19.784912 20286 ts_tablet_manager.cc:1428] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:19.960511 20224 raft_consensus.cc:482] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:19.960801 20224 raft_consensus.cc:504] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:19.961423 20224 leader_election.cc:288] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853), 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283)
I20260925 14:52:19.962045 20127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c432410f6e78454ab6651b594ce8cbbe" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6" is_pre_election: true
I20260925 14:52:19.962044 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c432410f6e78454ab6651b594ce8cbbe" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb41897f145a408b9d1289f9f5098bf7" is_pre_election: true
I20260925 14:52:19.962283 20127 raft_consensus.cc:2454] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 0.
I20260925 14:52:19.962365 19845 raft_consensus.cc:2454] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 0.
I20260925 14:52:19.962822 19917 leader_election.cc:302] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7496e357ecb34e188080263ab41816d6, acda07d8333f4ce6a5ea13190b6db950; no voters:
I20260925 14:52:19.963147 20224 raft_consensus.cc:2781] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:19.963256 20224 raft_consensus.cc:482] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:19.963382 20224 raft_consensus.cc:3037] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:19.965654 20224 raft_consensus.cc:504] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:19.966393 20224 leader_election.cc:288] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 election: Requested vote from peers eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853), 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283)
I20260925 14:52:19.966976 20127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c432410f6e78454ab6651b594ce8cbbe" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6"
I20260925 14:52:19.966958 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c432410f6e78454ab6651b594ce8cbbe" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb41897f145a408b9d1289f9f5098bf7"
I20260925 14:52:19.967204 20127 raft_consensus.cc:3037] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:19.967226 19845 raft_consensus.cc:3037] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:19.969498 20127 raft_consensus.cc:2454] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 1.
I20260925 14:52:19.969964 19917 leader_election.cc:302] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7496e357ecb34e188080263ab41816d6, acda07d8333f4ce6a5ea13190b6db950; no voters:
I20260925 14:52:19.970239 20224 raft_consensus.cc:2781] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:19.970424 20224 raft_consensus.cc:686] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 1 LEADER]: Becoming Leader. State: Replica: acda07d8333f4ce6a5ea13190b6db950, State: Running, Role: LEADER
I20260925 14:52:19.970669 19845 raft_consensus.cc:2454] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 1.
I20260925 14:52:19.970732 20224 consensus_queue.cc:237] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:19.974061 19687 catalog_manager.cc:5495] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 reported cstate change: term changed from 0 to 1, leader changed from <none> to acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194). New cstate: current_term: 1 leader_uuid: "acda07d8333f4ce6a5ea13190b6db950" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:20.036734 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:20.031586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.17.133.194:45347)
0925 14:52:20.032091 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:20.032095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:20.032407 (+ 312us) server_negotiation.cc:407] Connection header received
0925 14:52:20.032666 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:20.032670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:20.032725 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:20.032819 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:20.033228 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:20.033903 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:20.035066 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:20.035489 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:20.035612 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:20.036022 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:20.036114 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:20.036206 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:20.036336 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:52:20.036405 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":253,"threads_started":1}
I20260925 14:52:20.045670 20127 raft_consensus.cc:1261] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Refusing update from remote peer acda07d8333f4ce6a5ea13190b6db950: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:20.045841 19845 raft_consensus.cc:1261] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Refusing update from remote peer acda07d8333f4ce6a5ea13190b6db950: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:20.046268 20264 consensus_queue.cc:1035] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:20.046954 20264 consensus_queue.cc:1035] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:20.050508 20256 mvcc.cc:205] Tried to move back new op lower bound from 7333265162422325248 to 7333265162123177984. Current Snapshot: MvccSnapshot[applied={T|T < 7333265162422325248}]
I20260925 14:52:20.100019 19686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:20.100309 19686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:20.103227 19686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0: Sending DeleteTablet for 3 replicas of tablet c432410f6e78454ab6651b594ce8cbbe
I20260925 14:52:20.104131 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet c432410f6e78454ab6651b594ce8cbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:20 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:57200
I20260925 14:52:20.104125 19825 tablet_service.cc:1514] Processing DeleteTablet for tablet c432410f6e78454ab6651b594ce8cbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:20 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:59664
I20260925 14:52:20.104142 20102 tablet_service.cc:1514] Processing DeleteTablet for tablet c432410f6e78454ab6651b594ce8cbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:20 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:37794
I20260925 14:52:20.104382 20274 tablet_replica.cc:330] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: stopping tablet replica
I20260925 14:52:20.104400 20273 tablet_replica.cc:330] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: stopping tablet replica
I20260925 14:52:20.104568 20274 raft_consensus.cc:2229] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:20.104888 20274 raft_consensus.cc:2258] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:20.105041 20275 tablet_replica.cc:330] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: stopping tablet replica
I20260925 14:52:20.105047 20273 raft_consensus.cc:2229] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:20.105206 20273 raft_consensus.cc:2258] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:20.105233 20275 raft_consensus.cc:2229] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:20.105425 20275 raft_consensus.cc:2258] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:20.105301 19686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 14:52:20.105890 20274 ts_tablet_manager.cc:1905] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:20.106138 20273 ts_tablet_manager.cc:1905] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:52:20.106137 19686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:20.111334 20274 ts_tablet_manager.cc:1918] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:52:20.111531 20274 log.cc:1196] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/c432410f6e78454ab6651b594ce8cbbe
I20260925 14:52:20.111562 20273 ts_tablet_manager.cc:1918] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:52:20.111718 20273 log.cc:1196] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/c432410f6e78454ab6651b594ce8cbbe
I20260925 14:52:20.111946 20274 ts_tablet_manager.cc:1939] T c432410f6e78454ab6651b594ce8cbbe P acda07d8333f4ce6a5ea13190b6db950: Deleting consensus metadata
I20260925 14:52:20.112126 20273 ts_tablet_manager.cc:1939] T c432410f6e78454ab6651b594ce8cbbe P eb41897f145a408b9d1289f9f5098bf7: Deleting consensus metadata
I20260925 14:52:20.113230 19675 catalog_manager.cc:4841] TS acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194:45347): tablet c432410f6e78454ab6651b594ce8cbbe (table test-table [id=d30d1cf518914dbb841ca2a270058c7e]) successfully deleted
I20260925 14:52:20.113330 19964 tablet_service.cc:1467] Processing CreateTablet for tablet db85353bc434415ab988d499228bd5f8 (DEFAULT_TABLE table=test-table [id=0a9988edf3394fe69a3733a2bc720ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:20.113377 20102 tablet_service.cc:1467] Processing CreateTablet for tablet db85353bc434415ab988d499228bd5f8 (DEFAULT_TABLE table=test-table [id=0a9988edf3394fe69a3733a2bc720ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:20.113678 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db85353bc434415ab988d499228bd5f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:20.113758 20102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db85353bc434415ab988d499228bd5f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:20.117347 20275 ts_tablet_manager.cc:1905] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:20.118499 20286 tablet_bootstrap.cc:492] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: Bootstrap starting.
I20260925 14:52:20.118710 20285 tablet_bootstrap.cc:492] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: Bootstrap starting.
I20260925 14:52:20.120232 20285 tablet_bootstrap.cc:654] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:20.120528 20286 tablet_bootstrap.cc:654] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:20.122331 20285 tablet_bootstrap.cc:492] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: No bootstrap required, opened a new log
I20260925 14:52:20.122484 20285 ts_tablet_manager.cc:1397] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 14:52:20.122691 20286 tablet_bootstrap.cc:492] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: No bootstrap required, opened a new log
I20260925 14:52:20.122864 20286 ts_tablet_manager.cc:1397] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260925 14:52:20.123172 20285 raft_consensus.cc:348] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:20.123425 20285 raft_consensus.cc:374] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:20.123494 20285 raft_consensus.cc:729] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acda07d8333f4ce6a5ea13190b6db950, State: Initialized, Role: FOLLOWER
I20260925 14:52:20.123514 20286 raft_consensus.cc:348] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:20.123762 20286 raft_consensus.cc:374] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:20.123731 20285 consensus_queue.cc:260] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:20.123872 20286 raft_consensus.cc:729] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7496e357ecb34e188080263ab41816d6, State: Initialized, Role: FOLLOWER
I20260925 14:52:20.113641 19825 tablet_service.cc:1467] Processing CreateTablet for tablet db85353bc434415ab988d499228bd5f8 (DEFAULT_TABLE table=test-table [id=0a9988edf3394fe69a3733a2bc720ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:20.124245 20285 ts_tablet_manager.cc:1428] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 14:52:20.124328 19825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db85353bc434415ab988d499228bd5f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:20.124230 20286 consensus_queue.cc:260] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:20.125455 20286 ts_tablet_manager.cc:1428] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:52:20.123940 20275 ts_tablet_manager.cc:1918] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:52:20.125828 20275 log.cc:1196] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/c432410f6e78454ab6651b594ce8cbbe
I20260925 14:52:20.126267 20275 ts_tablet_manager.cc:1939] T c432410f6e78454ab6651b594ce8cbbe P 7496e357ecb34e188080263ab41816d6: Deleting consensus metadata
I20260925 14:52:20.127197 19674 catalog_manager.cc:4841] TS 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283): tablet c432410f6e78454ab6651b594ce8cbbe (table test-table [id=d30d1cf518914dbb841ca2a270058c7e]) successfully deleted
I20260925 14:52:20.129662 19672 catalog_manager.cc:4841] TS eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853): tablet c432410f6e78454ab6651b594ce8cbbe (table test-table [id=d30d1cf518914dbb841ca2a270058c7e]) successfully deleted
I20260925 14:52:20.130146 20284 tablet_bootstrap.cc:492] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: Bootstrap starting.
I20260925 14:52:20.131713 20284 tablet_bootstrap.cc:654] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:20.133136 20284 tablet_bootstrap.cc:492] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: No bootstrap required, opened a new log
I20260925 14:52:20.133296 20284 ts_tablet_manager.cc:1397] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 14:52:20.133728 20284 raft_consensus.cc:348] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:20.133868 20284 raft_consensus.cc:374] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:20.133916 20284 raft_consensus.cc:729] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb41897f145a408b9d1289f9f5098bf7, State: Initialized, Role: FOLLOWER
I20260925 14:52:20.134096 20284 consensus_queue.cc:260] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:20.134496 20284 ts_tablet_manager.cc:1428] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:20.190194 20224 raft_consensus.cc:482] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:20.190362 20224 raft_consensus.cc:504] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:20.190884 20224 leader_election.cc:288] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853), 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283)
I20260925 14:52:20.191377 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db85353bc434415ab988d499228bd5f8" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb41897f145a408b9d1289f9f5098bf7" is_pre_election: true
I20260925 14:52:20.191388 20127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db85353bc434415ab988d499228bd5f8" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6" is_pre_election: true
I20260925 14:52:20.191669 19845 raft_consensus.cc:2454] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 0.
I20260925 14:52:20.191687 20127 raft_consensus.cc:2454] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 0.
I20260925 14:52:20.192083 19915 leader_election.cc:302] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acda07d8333f4ce6a5ea13190b6db950, eb41897f145a408b9d1289f9f5098bf7; no voters:
I20260925 14:52:20.192386 20224 raft_consensus.cc:2781] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:20.192503 20224 raft_consensus.cc:482] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:20.192629 20224 raft_consensus.cc:3037] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:20.194739 20224 raft_consensus.cc:504] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:20.195199 20224 leader_election.cc:288] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 election: Requested vote from peers eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853), 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283)
I20260925 14:52:20.195647 20127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db85353bc434415ab988d499228bd5f8" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7496e357ecb34e188080263ab41816d6"
I20260925 14:52:20.195640 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db85353bc434415ab988d499228bd5f8" candidate_uuid: "acda07d8333f4ce6a5ea13190b6db950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb41897f145a408b9d1289f9f5098bf7"
I20260925 14:52:20.195830 20127 raft_consensus.cc:3037] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:20.195840 19845 raft_consensus.cc:3037] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:20.198539 19845 raft_consensus.cc:2454] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 1.
I20260925 14:52:20.198555 20127 raft_consensus.cc:2454] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acda07d8333f4ce6a5ea13190b6db950 in term 1.
I20260925 14:52:20.198896 19915 leader_election.cc:302] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7496e357ecb34e188080263ab41816d6, acda07d8333f4ce6a5ea13190b6db950, eb41897f145a408b9d1289f9f5098bf7; no voters:
I20260925 14:52:20.199162 20224 raft_consensus.cc:2781] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:20.199292 20224 raft_consensus.cc:686] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 1 LEADER]: Becoming Leader. State: Replica: acda07d8333f4ce6a5ea13190b6db950, State: Running, Role: LEADER
I20260925 14:52:20.199527 20224 consensus_queue.cc:237] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } }
I20260925 14:52:20.201916 19687 catalog_manager.cc:5495] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 reported cstate change: term changed from 0 to 1, leader changed from <none> to acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194). New cstate: current_term: 1 leader_uuid: "acda07d8333f4ce6a5ea13190b6db950" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acda07d8333f4ce6a5ea13190b6db950" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:20.233618 20127 raft_consensus.cc:1261] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Refusing update from remote peer acda07d8333f4ce6a5ea13190b6db950: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:20.233783 19845 raft_consensus.cc:1261] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Refusing update from remote peer acda07d8333f4ce6a5ea13190b6db950: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:20.234138 20224 consensus_queue.cc:1035] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7496e357ecb34e188080263ab41816d6" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:20.234400 20268 consensus_queue.cc:1035] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb41897f145a408b9d1289f9f5098bf7" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:52:20.252343 20299 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:52:21.830314 19687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:21.830605 19687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:21.834762 19687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 317e0a28e2c14ddcafdf9ef499517ad0: Sending DeleteTablet for 3 replicas of tablet db85353bc434415ab988d499228bd5f8
I20260925 14:52:21.835565 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet db85353bc434415ab988d499228bd5f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:21 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:57200
I20260925 14:52:21.835570 19825 tablet_service.cc:1514] Processing DeleteTablet for tablet db85353bc434415ab988d499228bd5f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:21 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:59664
I20260925 14:52:21.836000 20102 tablet_service.cc:1514] Processing DeleteTablet for tablet db85353bc434415ab988d499228bd5f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:21 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:37794
I20260925 14:52:21.836120 20328 tablet_replica.cc:330] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: stopping tablet replica
I20260925 14:52:21.836213 20329 tablet_replica.cc:330] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: stopping tablet replica
I20260925 14:52:21.836390 20328 raft_consensus.cc:2229] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:21.836524 20329 raft_consensus.cc:2229] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:21.836627 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19759
I20260925 14:52:21.836808 20328 raft_consensus.cc:2258] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:21.836905 20329 raft_consensus.cc:2258] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:21.837837 20328 ts_tablet_manager.cc:1905] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:21.842931 20329 ts_tablet_manager.cc:1905] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:21.847270 20328 ts_tablet_manager.cc:1918] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:52:21.847502 20328 log.cc:1196] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/db85353bc434415ab988d499228bd5f8
I20260925 14:52:21.848119 20328 ts_tablet_manager.cc:1939] T db85353bc434415ab988d499228bd5f8 P acda07d8333f4ce6a5ea13190b6db950: Deleting consensus metadata
I20260925 14:52:21.848644 20330 tablet_replica.cc:330] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: stopping tablet replica
I20260925 14:52:21.850365 19675 catalog_manager.cc:4841] TS acda07d8333f4ce6a5ea13190b6db950 (127.17.133.194:45347): tablet db85353bc434415ab988d499228bd5f8 (table test-table [id=0a9988edf3394fe69a3733a2bc720ec3]) successfully deleted
I20260925 14:52:21.850677 20330 raft_consensus.cc:2229] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:21.850972 20330 raft_consensus.cc:2258] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:21.852025 20330 ts_tablet_manager.cc:1905] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:21.852792 20329 ts_tablet_manager.cc:1918] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:52:21.852957 20329 log.cc:1196] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/db85353bc434415ab988d499228bd5f8
I20260925 14:52:21.853350 20329 ts_tablet_manager.cc:1939] T db85353bc434415ab988d499228bd5f8 P eb41897f145a408b9d1289f9f5098bf7: Deleting consensus metadata
I20260925 14:52:21.854357 19672 catalog_manager.cc:4841] TS eb41897f145a408b9d1289f9f5098bf7 (127.17.133.193:44853): tablet db85353bc434415ab988d499228bd5f8 (table test-table [id=0a9988edf3394fe69a3733a2bc720ec3]) successfully deleted
I20260925 14:52:21.858196 20330 ts_tablet_manager.cc:1918] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:52:21.858312 20330 log.cc:1196] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/db85353bc434415ab988d499228bd5f8
I20260925 14:52:21.858731 20330 ts_tablet_manager.cc:1939] T db85353bc434415ab988d499228bd5f8 P 7496e357ecb34e188080263ab41816d6: Deleting consensus metadata
I20260925 14:52:21.859738 19674 catalog_manager.cc:4841] TS 7496e357ecb34e188080263ab41816d6 (127.17.133.195:39283): tablet db85353bc434415ab988d499228bd5f8 (table test-table [id=0a9988edf3394fe69a3733a2bc720ec3]) successfully deleted
Sep 25 14:52:21 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:21.867915 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:21.838428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.17.133.193:44853)
0925 14:52:21.848622 (+ 10194us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:21.848630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:21.848653 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:52:21.848718 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:21.848724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:21.848817 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:21.848936 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:21.850635 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:21.852263 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:21.854326 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:21.855015 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:21.855159 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:21.862431 (+ 7272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:21.862465 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:21.862503 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:21.862549 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:21.865434 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:21.866076 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:21.866085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:21.866087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:21.866230 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:21.866534 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:21.866537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:21.866540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:21.866886 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:21.867114 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:21.867352 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 14:52:21.867517 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10033,"thread_start_us":9904,"threads_started":1}
W20260925 14:52:21.868559 19890 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33254
W20260925 14:52:21.869307 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19759: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:52:21.869496 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 19759
I20260925 14:52:21.892010 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19899
Sep 25 14:52:21 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:52:21.906270 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:21.892709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.17.133.194:45347)
0925 14:52:21.893242 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:21.893247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:21.893263 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:52:21.893315 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:21.893318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:21.893383 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:21.893471 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:21.894646 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:21.895337 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:21.896578 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:21.897125 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:21.897230 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:21.901241 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:21.901255 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:21.901271 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:21.901302 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:21.904048 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:21.904673 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:21.904680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:21.904683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:21.904792 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:21.905084 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:21.905087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:21.905089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:21.905400 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:21.905628 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:21.905852 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 14:52:21.906002 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":291,"threads_started":1}
W20260925 14:52:21.906977 20029 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156
W20260925 14:52:21.907532 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19899: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:52:21.907706 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 19899
I20260925 14:52:21.929160 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20038
Sep 25 14:52:21 dist-test-slave-v5vl krb5kdc[19643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:52:21.945297 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:21.929865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.17.133.195:39283)
0925 14:52:21.930385 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:21.930389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:21.930405 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:52:21.930454 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:21.930479 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:21.930537 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:21.930629 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:21.932255 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:21.933152 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:21.934891 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:21.935270 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:21.935359 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:21.939516 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:21.939529 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:21.939545 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:21.939575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:21.942414 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:21.943227 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:21.943240 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:21.943242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:21.943393 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:21.943774 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:21.943780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:21.943784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:21.944102 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:21.944421 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:21.944725 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 14:52:21.944931 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
W20260925 14:52:21.945956 20168 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54382
W20260925 14:52:21.946551 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20038: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:52:21.946717 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 20038
I20260925 14:52:21.963894 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 19656
I20260925 14:52:21.965131 19728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:22.078681 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 19656
2026-09-25T14:52:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (11158 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:22 dist-test-slave-v5vl krb5kdc[20338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:22 dist-test-slave-v5vl krb5kdc[20338](info): set up 2 sockets
Sep 25 14:52:22 dist-test-slave-v5vl krb5kdc[20338](info): commencing operation
krb5kdc: starting...
W20260925 14:52:23.149556 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:23 dist-test-slave-v5vl krb5kdc[20338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:52:23.331938 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46043
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:46043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:23.644804 20351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:23.645079 20351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:23.645188 20351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:23.652029 20351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:23.652112 20351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:23.652158 20351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:23.652206 20351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:23.652249 20351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:23.661410 20351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:46043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:23.662571 20351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:23.663779 20351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:23.675074 20356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:23.675112 20357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:23.675144 20359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:23.675755 20351 server_base.cc:1034] running on GCE node
I20260925 14:52:23.676560 20351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:23.678020 20351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:23.679219 20351 hybrid_clock.cc:648] HybridClock initialized: now 1790347943679162 us; error 52 us; skew 500 ppm
I20260925 14:52:23.679575 20351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:23 dist-test-slave-v5vl krb5kdc[20338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347943, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:23.682583 20351 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:52:23.683969 20351 webserver.cc:460] Webserver started at http://127.17.133.254:39311/ using document root <none> and password file <none>
I20260925 14:52:23.684741 20351 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:23.684885 20351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:23.685194 20351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:23.688166 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "17674496b7064b93aab918c0c34401c5"
format_stamp: "Formatted at 2026-09-25 14:52:23 on dist-test-slave-v5vl"
server_key: "2d09cb01e5888b43daf29c8794e35780"
server_key_iv: "c6ecb5c523a6641af38b4868c1b928bc"
server_key_version: "encryptionkey@0"
I20260925 14:52:23.688958 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "17674496b7064b93aab918c0c34401c5"
format_stamp: "Formatted at 2026-09-25 14:52:23 on dist-test-slave-v5vl"
server_key: "2d09cb01e5888b43daf29c8794e35780"
server_key_iv: "c6ecb5c523a6641af38b4868c1b928bc"
server_key_version: "encryptionkey@0"
I20260925 14:52:23.694021 20351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 14:52:23.697506 20366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:23.698853 20351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:52:23.699051 20351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "17674496b7064b93aab918c0c34401c5"
format_stamp: "Formatted at 2026-09-25 14:52:23 on dist-test-slave-v5vl"
server_key: "2d09cb01e5888b43daf29c8794e35780"
server_key_iv: "c6ecb5c523a6641af38b4868c1b928bc"
server_key_version: "encryptionkey@0"
I20260925 14:52:23.699259 20351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:23.713261 20351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:23.716964 20351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:23.717226 20351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:23.735180 20351 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:46043
I20260925 14:52:23.735211 20417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:46043 every 8 connection(s)
I20260925 14:52:23.736543 20351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:23.738106 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 20351
I20260925 14:52:23.738348 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:23.738965 17943 external_mini_cluster.cc:1428] Setting key 0723e12bcfa2a169f0d8b6adbec97daa
I20260925 14:52:23.740763 20418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 14:52:23 dist-test-slave-v5vl krb5kdc[20338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:23.754221 20418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5: Bootstrap starting.
I20260925 14:52:23.757555 20418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:23.758545 20418 log.cc:824] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:23.761848 20418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5: No bootstrap required, opened a new log
I20260925 14:52:23.761930 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:23.740996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.17.133.254:46043)
0925 14:52:23.741743 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:23.741758 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:23.741805 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:52:23.743890 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:23.743916 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:23.744427 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:23.744892 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:23.746116 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:23.748131 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:23.749387 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:23.750410 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:23.750592 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:23.754601 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:23.754608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:23.754622 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:23.754661 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:23.757549 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:23.758199 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:23.758205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:23.758210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:23.758356 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:23.758788 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:23.758793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:23.758796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:23.759492 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:23.759720 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:23.760236 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 14:52:23.760664 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":297,"threads_started":1}
I20260925 14:52:23.769114 20418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17674496b7064b93aab918c0c34401c5" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46043 } }
I20260925 14:52:23.769433 20418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:23.769502 20418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17674496b7064b93aab918c0c34401c5, State: Initialized, Role: FOLLOWER
I20260925 14:52:23.769960 20418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17674496b7064b93aab918c0c34401c5" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46043 } }
I20260925 14:52:23.770146 20418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:23.770236 20418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:23.770354 20418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:23.772313 20418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17674496b7064b93aab918c0c34401c5" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46043 } }
I20260925 14:52:23.772794 20418 leader_election.cc:302] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17674496b7064b93aab918c0c34401c5; no voters:
I20260925 14:52:23.773344 20418 leader_election.cc:288] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:23.773468 20423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:23.774711 20423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [term 1 LEADER]: Becoming Leader. State: Replica: 17674496b7064b93aab918c0c34401c5, State: Running, Role: LEADER
I20260925 14:52:23.775305 20423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17674496b7064b93aab918c0c34401c5" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46043 } }
I20260925 14:52:23.775444 20418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:23.778239 20425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17674496b7064b93aab918c0c34401c5. Latest consensus state: current_term: 1 leader_uuid: "17674496b7064b93aab918c0c34401c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17674496b7064b93aab918c0c34401c5" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46043 } } }
I20260925 14:52:23.778465 20424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17674496b7064b93aab918c0c34401c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17674496b7064b93aab918c0c34401c5" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46043 } } }
I20260925 14:52:23.778669 20425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:23.778754 20424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:23.779572 20432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:23.784914 20432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:23.793146 20432 catalog_manager.cc:1347] Generated new cluster ID: cb5136b30e464cf8b33d2e055108fd40
I20260925 14:52:23.793242 20432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:23.814186 20432 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:23.815966 20432 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:23.829010 20432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17674496b7064b93aab918c0c34401c5: Generated new TSK 0
I20260925 14:52:23.830016 20432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:52:23.884331 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46043
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:24.208079 20444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:24.208365 20444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:24.208526 20444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:24.215580 20444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:24.215732 20444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:24.215910 20444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:24.226009 20444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:24.227238 20444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:24.228487 20444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:24.240698 20450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:24.240751 20449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:24.240808 20452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:24.241389 20444 server_base.cc:1034] running on GCE node
I20260925 14:52:24.241991 20444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:24.242846 20444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:24.244062 20444 hybrid_clock.cc:648] HybridClock initialized: now 1790347944244030 us; error 47 us; skew 500 ppm
I20260925 14:52:24.244350 20444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:24 dist-test-slave-v5vl krb5kdc[20338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347944, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:24.247306 20444 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:52:24.248694 20444 webserver.cc:460] Webserver started at http://127.17.133.193:33357/ using document root <none> and password file <none>
I20260925 14:52:24.249279 20444 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:24.249370 20444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:24.249610 20444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:24.252871 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "68cf07b21d2b45b19784f1c5eabda059"
format_stamp: "Formatted at 2026-09-25 14:52:24 on dist-test-slave-v5vl"
server_key: "d15ff561b649ed7ea4affc218326f6a0"
server_key_iv: "9d960c0d7bbd2b26527bd6d0a3a470be"
server_key_version: "encryptionkey@0"
I20260925 14:52:24.253594 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "68cf07b21d2b45b19784f1c5eabda059"
format_stamp: "Formatted at 2026-09-25 14:52:24 on dist-test-slave-v5vl"
server_key: "d15ff561b649ed7ea4affc218326f6a0"
server_key_iv: "9d960c0d7bbd2b26527bd6d0a3a470be"
server_key_version: "encryptionkey@0"
I20260925 14:52:24.258128 20444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:52:24.261258 20459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:24.262677 20444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:24.262785 20444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "68cf07b21d2b45b19784f1c5eabda059"
format_stamp: "Formatted at 2026-09-25 14:52:24 on dist-test-slave-v5vl"
server_key: "d15ff561b649ed7ea4affc218326f6a0"
server_key_iv: "9d960c0d7bbd2b26527bd6d0a3a470be"
server_key_version: "encryptionkey@0"
I20260925 14:52:24.262931 20444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:24.288084 20444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:24.292472 20444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:24.292783 20444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:24.293655 20444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:24.294858 20444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:24.294942 20444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:24.295029 20444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:24.295073 20444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:24.327039 20444 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:39043
I20260925 14:52:24.327090 20571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:39043 every 8 connection(s)
I20260925 14:52:24.328413 20444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:24.332517 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 20444
I20260925 14:52:24.332731 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:24.333535 17943 external_mini_cluster.cc:1428] Setting key fb75df4b9c63c7548e85d60ba90cdc8a
Sep 25 14:52:24 dist-test-slave-v5vl krb5kdc[20338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347944, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:24.351064 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:24.330956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:34649 (local address 127.17.133.254:46043)
0925 14:52:24.331391 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:24.331398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:24.332406 (+ 1008us) server_negotiation.cc:407] Connection header received
0925 14:52:24.334893 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:24.334899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:24.334973 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:24.335083 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:24.337046 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:24.337722 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:24.340299 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:24.340812 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:24.340968 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:24.344209 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:24.344218 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:24.344226 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:24.344271 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:24.347812 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:24.348530 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:24.348538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:24.348546 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:24.348761 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:24.349157 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:24.349164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:24.349171 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:24.349603 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:24.349776 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:24.350421 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 14:52:24.350642 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":193,"threads_started":1}
I20260925 14:52:24.351806 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:24.331491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:46043 (local address 127.17.133.193:34649)
0925 14:52:24.332219 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:24.332275 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:24.334639 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:24.335336 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:24.335344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:24.336043 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:24.336876 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:24.336903 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:24.337915 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:24.337923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:24.340056 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:24.340091 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:24.341027 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:24.341032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:24.341237 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:24.341908 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:24.341931 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:24.343999 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:24.347950 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:24.347957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:24.347969 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:24.348388 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:24.348873 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:24.348877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:24.348879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:24.349049 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:24.349731 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:24.349737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:24.350071 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 14:52:24.350289 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 14:52:24.350649 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":253,"threads_started":1}
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:52:24.353806 20572 heartbeater.cc:347] Connected to a master server at 127.17.133.254:46043
I20260925 14:52:24.354172 20572 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:24.355126 20572 heartbeater.cc:511] Master 127.17.133.254:46043 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
I20260925 14:52:24.357314 20383 ts_manager.cc:194] Registered new tserver with Master: 68cf07b21d2b45b19784f1c5eabda059 (127.17.133.193:39043)
I20260925 14:52:24.358732 20383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:34649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:52:24.379504 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46043
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:24.694067 20578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:24.694324 20578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:24.694491 20578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:24.701550 20578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:24.701676 20578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:24.701846 20578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:24.710953 20578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:24.712157 20578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:24.713405 20578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:24.726142 20584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:24.726083 20583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:24.726190 20586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:24.727147 20578 server_base.cc:1034] running on GCE node
I20260925 14:52:24.727815 20578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:24.728814 20578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:24.730050 20578 hybrid_clock.cc:648] HybridClock initialized: now 1790347944729992 us; error 49 us; skew 500 ppm
I20260925 14:52:24.730360 20578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:24 dist-test-slave-v5vl krb5kdc[20338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347944, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:24.733361 20578 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:52:24.734972 20578 webserver.cc:460] Webserver started at http://127.17.133.194:34443/ using document root <none> and password file <none>
I20260925 14:52:24.735652 20578 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:24.735791 20578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:24.736100 20578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:24.738901 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "452f4d65c7264dc498bd611a1c964585"
format_stamp: "Formatted at 2026-09-25 14:52:24 on dist-test-slave-v5vl"
server_key: "b8c10f9ac30d62f02fd1304b5c93fb44"
server_key_iv: "7d05a9e6cfdd76a3d3e904c5d3eedf02"
server_key_version: "encryptionkey@0"
I20260925 14:52:24.739675 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "452f4d65c7264dc498bd611a1c964585"
format_stamp: "Formatted at 2026-09-25 14:52:24 on dist-test-slave-v5vl"
server_key: "b8c10f9ac30d62f02fd1304b5c93fb44"
server_key_iv: "7d05a9e6cfdd76a3d3e904c5d3eedf02"
server_key_version: "encryptionkey@0"
I20260925 14:52:24.745033 20578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 14:52:24.748500 20593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:24.749979 20578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:52:24.750106 20578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "452f4d65c7264dc498bd611a1c964585"
format_stamp: "Formatted at 2026-09-25 14:52:24 on dist-test-slave-v5vl"
server_key: "b8c10f9ac30d62f02fd1304b5c93fb44"
server_key_iv: "7d05a9e6cfdd76a3d3e904c5d3eedf02"
server_key_version: "encryptionkey@0"
I20260925 14:52:24.750267 20578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:24.787662 20578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:24.791563 20578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:24.791805 20578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:24.792744 20578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:24.793998 20578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:24.794072 20578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:24.794152 20578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:24.794195 20578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:24.828258 20578 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:42793
I20260925 14:52:24.828331 20705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:42793 every 8 connection(s)
I20260925 14:52:24.829571 20578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:24.836476 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 20578
I20260925 14:52:24.836683 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:24.837240 17943 external_mini_cluster.cc:1428] Setting key 92eb25b0e92748da05fb1a6176b9d16e
Sep 25 14:52:24 dist-test-slave-v5vl krb5kdc[20338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347944, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:24.849800 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:24.832018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:38621 (local address 127.17.133.254:46043)
0925 14:52:24.832188 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:24.832193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:24.833382 (+ 1189us) server_negotiation.cc:407] Connection header received
0925 14:52:24.835141 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:24.835145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:24.835217 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:24.835305 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:24.837004 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:24.837990 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:24.839806 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:24.840247 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:24.840381 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:24.844038 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:24.844043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:24.844046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:24.844078 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:24.846649 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:24.847376 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:24.847386 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:24.847388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:24.847509 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:24.847956 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:24.847965 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:24.847967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:24.848226 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:24.848378 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:24.849192 (+ 814us) server_negotiation.cc:299] Negotiation successful
0925 14:52:24.849417 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:52:24.850828 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:24.832495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:46043 (local address 127.17.133.194:38621)
0925 14:52:24.833223 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:24.833260 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:24.834873 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:24.835563 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:24.835572 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:24.836088 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:24.836826 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:24.836852 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:24.838140 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:24.838145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:24.839663 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:24.839674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:24.840477 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:24.840484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:24.840792 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:24.841476 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:24.841498 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:24.843806 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:24.846794 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:24.846803 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:24.846823 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:24.847244 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:24.847622 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:24.847627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:24.847631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:24.847832 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:24.848356 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:24.848364 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:24.848795 (+ 431us) client_negotiation.cc:769] Sending connection context
0925 14:52:24.849070 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 14:52:24.849622 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":263,"threads_started":1}
I20260925 14:52:24.852334 20706 heartbeater.cc:347] Connected to a master server at 127.17.133.254:46043
I20260925 14:52:24.852717 20706 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:24.853665 20706 heartbeater.cc:511] Master 127.17.133.254:46043 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
I20260925 14:52:24.855378 20383 ts_manager.cc:194] Registered new tserver with Master: 452f4d65c7264dc498bd611a1c964585 (127.17.133.194:42793)
I20260925 14:52:24.856221 20383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:38621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:52:24.879639 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46043
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:25.192036 20711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:25.192332 20711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:25.192500 20711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:25.199384 20711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:25.199524 20711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:25.199710 20711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:25.209293 20711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:25.210552 20711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:25.211817 20711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:25.225064 20719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:25.225124 20716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:25.225119 20717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:25.225939 20711 server_base.cc:1034] running on GCE node
I20260925 14:52:25.226433 20711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:25.227250 20711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:25.228437 20711 hybrid_clock.cc:648] HybridClock initialized: now 1790347945228378 us; error 50 us; skew 500 ppm
I20260925 14:52:25.228773 20711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:25 dist-test-slave-v5vl krb5kdc[20338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347945, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:25.231719 20711 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:52:25.233175 20711 webserver.cc:460] Webserver started at http://127.17.133.195:39883/ using document root <none> and password file <none>
I20260925 14:52:25.233829 20711 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:25.233970 20711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:25.234256 20711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:25.237176 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "804029ab2a894da6ba9350d55e725cb2"
format_stamp: "Formatted at 2026-09-25 14:52:25 on dist-test-slave-v5vl"
server_key: "d0b49c6800d58f01fe977399fedcc7c7"
server_key_iv: "86bd409b0d721379f027d725e695c9ae"
server_key_version: "encryptionkey@0"
I20260925 14:52:25.237943 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "804029ab2a894da6ba9350d55e725cb2"
format_stamp: "Formatted at 2026-09-25 14:52:25 on dist-test-slave-v5vl"
server_key: "d0b49c6800d58f01fe977399fedcc7c7"
server_key_iv: "86bd409b0d721379f027d725e695c9ae"
server_key_version: "encryptionkey@0"
I20260925 14:52:25.242952 20711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:52:25.246284 20726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:25.247562 20711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:52:25.247735 20711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "804029ab2a894da6ba9350d55e725cb2"
format_stamp: "Formatted at 2026-09-25 14:52:25 on dist-test-slave-v5vl"
server_key: "d0b49c6800d58f01fe977399fedcc7c7"
server_key_iv: "86bd409b0d721379f027d725e695c9ae"
server_key_version: "encryptionkey@0"
I20260925 14:52:25.247941 20711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:25.259490 20711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:25.263012 20711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:25.263300 20711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:25.264149 20711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:25.265398 20711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:25.265493 20711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:25.265633 20711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:25.265702 20711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:25.297475 20711 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:45985
I20260925 14:52:25.297518 20838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:45985 every 8 connection(s)
I20260925 14:52:25.298686 20711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:25.305907 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 20711
I20260925 14:52:25.306074 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:25.306476 17943 external_mini_cluster.cc:1428] Setting key fa9eb6422affa52bd4bd59b3d4f6eded
Sep 25 14:52:25 dist-test-slave-v5vl krb5kdc[20338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347945, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:25.317250 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:25.301122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:54155 (local address 127.17.133.254:46043)
0925 14:52:25.301294 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:25.301298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:25.302385 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 14:52:25.304368 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:25.304375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:25.304458 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:25.304562 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:25.305971 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:25.306834 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:25.308688 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:25.309179 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:25.309297 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:25.312360 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:25.312364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:25.312368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:25.312396 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:25.314817 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:25.315466 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:25.315470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:25.315472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:25.315581 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:25.315910 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:25.315913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:25.315915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:25.316142 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:25.316241 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:25.316888 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 14:52:25.317025 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:52:25.318099 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:25.301540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:46043 (local address 127.17.133.195:54155)
0925 14:52:25.302208 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:25.302245 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:25.304011 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:25.304805 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:25.304812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:25.305270 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:25.305811 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:25.305827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:25.306981 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:25.306987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:25.308546 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:25.308557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:25.309383 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:25.309389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:25.309639 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:25.310211 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:25.310230 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:25.312193 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:25.314942 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:25.314948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:25.314958 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:25.315374 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:25.315667 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:25.315670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:25.315672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:25.315825 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:25.316237 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:25.316242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:25.316509 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 14:52:25.316805 (+ 296us) client_negotiation.cc:240] Negotiation successful
0925 14:52:25.317134 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":266,"threads_started":1}
I20260925 14:52:25.319350 20839 heartbeater.cc:347] Connected to a master server at 127.17.133.254:46043
I20260925 14:52:25.319674 20839 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:25.320479 20839 heartbeater.cc:511] Master 127.17.133.254:46043 requested a full tablet report, sending...
I20260925 14:52:25.322038 20383 ts_manager.cc:194] Registered new tserver with Master: 804029ab2a894da6ba9350d55e725cb2 (127.17.133.195:45985)
I20260925 14:52:25.322825 20383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:54155
I20260925 14:52:25.333312 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:25.361616 20572 heartbeater.cc:503] Master 127.17.133.254:46043 was elected leader, sending a full tablet report...
W20260925 14:52:25.363965 20574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:25.361314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.17.133.254:46043)
0925 14:52:25.361520 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:25.361526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:25.362048 (+ 522us) server_negotiation.cc:407] Connection header received
0925 14:52:25.362265 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:25.362269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:25.362333 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:25.362463 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:25.363708 (+ 1245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52684: BlockingRecv error: recv got EOF from 127.0.0.1:52684 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:52:25.366974 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20444
W20260925 14:52:25.369422 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20444: Not authorized: Client connection negotiation failed: client connection to 127.17.133.193:39043: server requires authentication, but client does not have Kerberos credentials available
I20260925 14:52:25.369634 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 20444
I20260925 14:52:25.385434 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20578
W20260925 14:52:25.387835 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20578: Not authorized: Client connection negotiation failed: client connection to 127.17.133.194:42793: server requires authentication, but client does not have Kerberos credentials available
I20260925 14:52:25.387935 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 20578
I20260925 14:52:25.403079 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20711
W20260925 14:52:25.406056 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20711: Not authorized: Client connection negotiation failed: client connection to 127.17.133.195:45985: server requires authentication, but client does not have Kerberos credentials available
I20260925 14:52:25.406227 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 20711
I20260925 14:52:25.421339 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20351
I20260925 14:52:25.422567 20413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:25.523563 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 20351
2026-09-25T14:52:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3441 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:25 dist-test-slave-v5vl krb5kdc[20856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:25 dist-test-slave-v5vl krb5kdc[20856](info): set up 2 sockets
Sep 25 14:52:25 dist-test-slave-v5vl krb5kdc[20856](info): commencing operation
krb5kdc: starting...
W20260925 14:52:26.601594 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.010s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:26 dist-test-slave-v5vl krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:52:26.783532 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:41191
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:41191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:27.096451 20869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:27.096829 20869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:27.096998 20869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:27.103679 20869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:27.103829 20869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:27.103942 20869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:27.104053 20869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:27.104143 20869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:27.113894 20869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:41191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:41191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:27.115242 20869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:27.116573 20869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:27.128708 20875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:27.128688 20877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:27.128679 20874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:27.129732 20869 server_base.cc:1034] running on GCE node
I20260925 14:52:27.130527 20869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:27.131887 20869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:27.133106 20869 hybrid_clock.cc:648] HybridClock initialized: now 1790347947133069 us; error 51 us; skew 500 ppm
I20260925 14:52:27.133457 20869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:27 dist-test-slave-v5vl krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347947, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:27.139581 20869 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:52:27.141126 20869 webserver.cc:460] Webserver started at http://127.17.133.254:34471/ using document root <none> and password file <none>
I20260925 14:52:27.141767 20869 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:27.141865 20869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:27.142112 20869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:27.145038 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493"
format_stamp: "Formatted at 2026-09-25 14:52:27 on dist-test-slave-v5vl"
server_key: "1c059eb559a9d1b4cbda2abd12f67a5a"
server_key_iv: "8bbe4ee98e7ce15702f10077d184aef6"
server_key_version: "encryptionkey@0"
I20260925 14:52:27.145778 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493"
format_stamp: "Formatted at 2026-09-25 14:52:27 on dist-test-slave-v5vl"
server_key: "1c059eb559a9d1b4cbda2abd12f67a5a"
server_key_iv: "8bbe4ee98e7ce15702f10077d184aef6"
server_key_version: "encryptionkey@0"
I20260925 14:52:27.150637 20869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:52:27.153872 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:27.155122 20869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:52:27.155258 20869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493"
format_stamp: "Formatted at 2026-09-25 14:52:27 on dist-test-slave-v5vl"
server_key: "1c059eb559a9d1b4cbda2abd12f67a5a"
server_key_iv: "8bbe4ee98e7ce15702f10077d184aef6"
server_key_version: "encryptionkey@0"
I20260925 14:52:27.155419 20869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:27.177513 20869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:27.181795 20869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:27.182026 20869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:27.199805 20869 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:41191
I20260925 14:52:27.199826 20935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:41191 every 8 connection(s)
I20260925 14:52:27.201114 20869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:27.202680 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 20869
I20260925 14:52:27.202913 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:27.203533 17943 external_mini_cluster.cc:1428] Setting key 362fb49f7383fb9ee1f0009738dc5070
I20260925 14:52:27.205225 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 14:52:27 dist-test-slave-v5vl krb5kdc[20856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:27.218257 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493: Bootstrap starting.
I20260925 14:52:27.222129 20936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:27.223347 20936 log.cc:824] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:27.226547 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493: No bootstrap required, opened a new log
I20260925 14:52:27.226840 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:27.205387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.17.133.254:41191)
0925 14:52:27.206169 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:27.206187 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:27.206235 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:52:27.208294 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:27.208321 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:27.208855 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:27.209300 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:27.210547 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:27.212618 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:27.213878 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:27.214899 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:27.215102 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:27.218866 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:27.218875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:27.218893 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:27.218931 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:27.222659 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:27.223279 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:27.223288 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:27.223298 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:27.223519 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:27.223912 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:27.223919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:27.223924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:27.224494 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:27.224860 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:27.225245 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 14:52:27.225635 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":317,"threads_started":1}
I20260925 14:52:27.230639 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:52:27.234534 20936 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41191 } }
I20260925 14:52:27.234870 20936 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:27.234987 20936 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c8e48f409a4aacbc5c98ab3b1b0493, State: Initialized, Role: FOLLOWER
I20260925 14:52:27.235574 20936 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41191 } }
I20260925 14:52:27.235801 20936 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:27.235924 20936 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:27.236112 20936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:27.238219 20936 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41191 } }
I20260925 14:52:27.238682 20936 leader_election.cc:302] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2c8e48f409a4aacbc5c98ab3b1b0493; no voters:
I20260925 14:52:27.239332 20936 leader_election.cc:288] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:27.239439 20942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:27.240485 20942 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [term 1 LEADER]: Becoming Leader. State: Replica: f2c8e48f409a4aacbc5c98ab3b1b0493, State: Running, Role: LEADER
I20260925 14:52:27.241068 20942 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41191 } }
I20260925 14:52:27.241516 20936 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:27.243799 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2c8e48f409a4aacbc5c98ab3b1b0493. Latest consensus state: current_term: 1 leader_uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41191 } } }
I20260925 14:52:27.244196 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:27.244654 20943 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8e48f409a4aacbc5c98ab3b1b0493" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41191 } } }
I20260925 14:52:27.244983 20943 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:27.247481 20950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:27.253014 20950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:27.269307 20950 catalog_manager.cc:1347] Generated new cluster ID: 931f4e49dd3b46d7879266c9583d6ff3
I20260925 14:52:27.269482 20950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:27.283610 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:27.274658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.17.133.254:41191)
0925 14:52:27.274847 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:27.274854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:27.277171 (+ 2317us) server_negotiation.cc:407] Connection header received
0925 14:52:27.277386 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:27.277391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:27.277450 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:27.277553 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:27.278512 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:27.279266 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:27.280523 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:27.281009 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:27.281144 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:27.282180 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:27.282191 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:27.282194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:27.282759 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:27.282819 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:27.282848 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:27.282982 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:27.283127 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 14:52:27.283212 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:52:27.285497 20950 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:27.287227 20950 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:27.294407 20950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2c8e48f409a4aacbc5c98ab3b1b0493: Generated new TSK 0
I20260925 14:52:27.295147 20950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:52:27.307216 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20869
I20260925 14:52:27.308817 20931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:27.398514 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 20869
2026-09-25T14:52:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1869 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:27 dist-test-slave-v5vl krb5kdc[20970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:27 dist-test-slave-v5vl krb5kdc[20970](info): set up 2 sockets
Sep 25 14:52:27 dist-test-slave-v5vl krb5kdc[20970](info): commencing operation
krb5kdc: starting...
W20260925 14:52:28.461377 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:28 dist-test-slave-v5vl krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:52:28.641597 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:33725
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:33725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:28.945801 20983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:28.946074 20983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:28.946168 20983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:28.953145 20983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:28.953239 20983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:28.953284 20983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:28.953339 20983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:28.953382 20983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:28.962461 20983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:33725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:33725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:28.963672 20983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:28.964928 20983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:28.976145 20988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:28.976184 20989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:28.976120 20991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:28.977298 20983 server_base.cc:1034] running on GCE node
I20260925 14:52:28.977994 20983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:28.979259 20983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:28.980473 20983 hybrid_clock.cc:648] HybridClock initialized: now 1790347948980433 us; error 59 us; skew 500 ppm
I20260925 14:52:28.980808 20983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:28 dist-test-slave-v5vl krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347948, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:28.983752 20983 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:52:28.985247 20983 webserver.cc:460] Webserver started at http://127.17.133.254:33893/ using document root <none> and password file <none>
I20260925 14:52:28.985818 20983 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:28.985927 20983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:28.986168 20983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:28.989342 20983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf"
format_stamp: "Formatted at 2026-09-25 14:52:28 on dist-test-slave-v5vl"
server_key: "edae78222343b1d02ea40817873d2f10"
server_key_iv: "e6e49d7245a959c1348819450f26a77e"
server_key_version: "encryptionkey@0"
I20260925 14:52:28.990001 20983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf"
format_stamp: "Formatted at 2026-09-25 14:52:28 on dist-test-slave-v5vl"
server_key: "edae78222343b1d02ea40817873d2f10"
server_key_iv: "e6e49d7245a959c1348819450f26a77e"
server_key_version: "encryptionkey@0"
I20260925 14:52:28.994740 20983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:52:28.997947 20998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:28.999271 20983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:52:28.999449 20983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf"
format_stamp: "Formatted at 2026-09-25 14:52:28 on dist-test-slave-v5vl"
server_key: "edae78222343b1d02ea40817873d2f10"
server_key_iv: "e6e49d7245a959c1348819450f26a77e"
server_key_version: "encryptionkey@0"
I20260925 14:52:28.999644 20983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:29.040645 20983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:29.044629 20983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:29.044937 20983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:29.061978 20983 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:33725
I20260925 14:52:29.062027 21049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:33725 every 8 connection(s)
I20260925 14:52:29.063207 20983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:29.066738 21050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:29.068674 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 20983
I20260925 14:52:29.068905 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:29.069671 17943 external_mini_cluster.cc:1428] Setting key c784520809699bfa048e223dad17053a
I20260925 14:52:29.079017 21050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf: Bootstrap starting.
I20260925 14:52:29.082681 21050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf: Neither blocks nor log segments found. Creating new log.
Sep 25 14:52:29 dist-test-slave-v5vl krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:29.083877 21050 log.cc:824] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:29.087505 21050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf: No bootstrap required, opened a new log
I20260925 14:52:29.096256 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:29.071626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.17.133.254:33725)
0925 14:52:29.072438 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:29.072449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:29.072488 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:52:29.074166 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:29.074187 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:29.074559 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:29.074851 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:29.076516 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:29.077813 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:29.079475 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:29.080085 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:29.080213 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:29.084954 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:29.084965 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:29.084989 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:29.085042 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:29.091585 (+ 6543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:29.092510 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:29.092519 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:29.092528 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:29.092764 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:29.093273 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:29.093279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:29.093282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:29.094047 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:29.094364 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:29.094810 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 14:52:29.095233 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":378,"threads_started":1}
I20260925 14:52:29.097731 21050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33725 } }
I20260925 14:52:29.098085 21050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:29.098207 21050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef4ea6ab98e4a48a43521ba4cfa5adf, State: Initialized, Role: FOLLOWER
I20260925 14:52:29.098778 21050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33725 } }
I20260925 14:52:29.099004 21050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:29.099149 21050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:29.099320 21050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:29.101481 21050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33725 } }
I20260925 14:52:29.101958 21050 leader_election.cc:302] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ef4ea6ab98e4a48a43521ba4cfa5adf; no voters:
I20260925 14:52:29.102543 21050 leader_election.cc:288] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:29.102684 21055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:29.103461 21055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [term 1 LEADER]: Becoming Leader. State: Replica: 8ef4ea6ab98e4a48a43521ba4cfa5adf, State: Running, Role: LEADER
I20260925 14:52:29.104322 21055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33725 } }
I20260925 14:52:29.104885 21050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:29.107235 21056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33725 } } }
I20260925 14:52:29.107200 21057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ef4ea6ab98e4a48a43521ba4cfa5adf. Latest consensus state: current_term: 1 leader_uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef4ea6ab98e4a48a43521ba4cfa5adf" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33725 } } }
I20260925 14:52:29.107600 21056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:29.107648 21057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:29.108711 21064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:29.114780 21064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:29.121997 21064 catalog_manager.cc:1347] Generated new cluster ID: 481efcfa99fe4c19b72dfe1da633129a
I20260925 14:52:29.122089 21064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:29.138183 21064 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:29.140024 21064 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:29.148448 21064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ef4ea6ab98e4a48a43521ba4cfa5adf: Generated new TSK 0
I20260925 14:52:29.149524 21064 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:52:29.196226 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33725
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:29.512215 21076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:29.512483 21076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:29.512672 21076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:29.519748 21076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:29.519869 21076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:29.520040 21076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:29.529711 21076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:29.530973 21076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:29.532260 21076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:29.545351 21081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:29.545516 21084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:29.545311 21082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:29.546569 21076 server_base.cc:1034] running on GCE node
I20260925 14:52:29.547138 21076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:29.548112 21076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:29.549332 21076 hybrid_clock.cc:648] HybridClock initialized: now 1790347949549284 us; error 38 us; skew 500 ppm
I20260925 14:52:29.549692 21076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:29 dist-test-slave-v5vl krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347949, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:29.552824 21076 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:52:29.554234 21076 webserver.cc:460] Webserver started at http://127.17.133.193:38511/ using document root <none> and password file <none>
I20260925 14:52:29.554929 21076 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:29.555070 21076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:29.555369 21076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:29.558223 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "ced60ecd220c49688c08e1f6402fdc4d"
format_stamp: "Formatted at 2026-09-25 14:52:29 on dist-test-slave-v5vl"
server_key: "3c887c7b32865406888c778a9c42a260"
server_key_iv: "7c1c7949887401f9d51525f5da01c459"
server_key_version: "encryptionkey@0"
I20260925 14:52:29.558990 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "ced60ecd220c49688c08e1f6402fdc4d"
format_stamp: "Formatted at 2026-09-25 14:52:29 on dist-test-slave-v5vl"
server_key: "3c887c7b32865406888c778a9c42a260"
server_key_iv: "7c1c7949887401f9d51525f5da01c459"
server_key_version: "encryptionkey@0"
I20260925 14:52:29.563838 21076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 14:52:29.567176 21091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:29.568446 21076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:52:29.568673 21076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "ced60ecd220c49688c08e1f6402fdc4d"
format_stamp: "Formatted at 2026-09-25 14:52:29 on dist-test-slave-v5vl"
server_key: "3c887c7b32865406888c778a9c42a260"
server_key_iv: "7c1c7949887401f9d51525f5da01c459"
server_key_version: "encryptionkey@0"
I20260925 14:52:29.568871 21076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:29.598492 21076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:29.602458 21076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:29.602737 21076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:29.603729 21076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:29.605176 21076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:29.605268 21076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:29.605417 21076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:29.605484 21076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:29.637454 21076 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:46693
I20260925 14:52:29.637494 21203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:46693 every 8 connection(s)
I20260925 14:52:29.638702 21076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:29.644492 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 21076
I20260925 14:52:29.644755 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:29.645602 17943 external_mini_cluster.cc:1428] Setting key 16a2565118ac7e2ca2a65da0b668884a
Sep 25 14:52:29 dist-test-slave-v5vl krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347949, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:29.661437 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:29.641408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:43205 (local address 127.17.133.254:33725)
0925 14:52:29.641867 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:29.641873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:29.642799 (+ 926us) server_negotiation.cc:407] Connection header received
0925 14:52:29.644756 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:29.644768 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:29.644838 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:29.644987 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:29.646919 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:29.647908 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:29.650531 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:29.650955 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:29.651116 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:29.655618 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:29.655623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:29.655626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:29.655654 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:29.658158 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:29.659070 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:29.659073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:29.659076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:29.659184 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:29.659672 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:29.659675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:29.659677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:29.659930 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:29.660044 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:29.660976 (+ 932us) server_negotiation.cc:299] Negotiation successful
0925 14:52:29.661154 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":338,"thread_start_us":224,"threads_started":1}
I20260925 14:52:29.663045 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:29.641987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33725 (local address 127.17.133.193:43205)
0925 14:52:29.642640 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:29.642682 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:29.644392 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:29.645256 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:29.645269 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:29.646006 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:29.646760 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:29.646779 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:29.648078 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:29.648085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:29.650392 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:29.650408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:29.651200 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:29.651209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:29.651535 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:29.652233 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:29.652265 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:29.655368 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:29.658312 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:29.658323 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:29.658344 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:29.658955 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:29.659290 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:29.659298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:29.659304 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:29.659563 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:29.660058 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:29.660069 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:29.660552 (+ 483us) client_negotiation.cc:769] Sending connection context
0925 14:52:29.660891 (+ 339us) client_negotiation.cc:240] Negotiation successful
0925 14:52:29.661499 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":259,"threads_started":1}
I20260925 14:52:29.664691 21204 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33725
I20260925 14:52:29.664996 21204 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:29.665710 21204 heartbeater.cc:511] Master 127.17.133.254:33725 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:52:29.667927 21015 ts_manager.cc:194] Registered new tserver with Master: ced60ecd220c49688c08e1f6402fdc4d (127.17.133.193:46693)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
I20260925 14:52:29.670128 21015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:43205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:52:29.691004 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33725
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:30.002231 21210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:30.002604 21210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:30.002779 21210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:30.010583 21210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:30.010751 21210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:30.010934 21210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:30.020813 21210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:30.022171 21210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:30.023525 21210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:30.036984 21215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:30.037119 21216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:30.037022 21218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:30.037981 21210 server_base.cc:1034] running on GCE node
I20260925 14:52:30.038550 21210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:30.039443 21210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:30.040688 21210 hybrid_clock.cc:648] HybridClock initialized: now 1790347950040642 us; error 49 us; skew 500 ppm
I20260925 14:52:30.040989 21210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347950, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:30.043987 21210 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:52:30.045389 21210 webserver.cc:460] Webserver started at http://127.17.133.194:33261/ using document root <none> and password file <none>
I20260925 14:52:30.045981 21210 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:30.046077 21210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:30.046386 21210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:30.049185 21210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "21f69003df1f46d69b32f3dea449d7bf"
format_stamp: "Formatted at 2026-09-25 14:52:30 on dist-test-slave-v5vl"
server_key: "08e9249af40be4ddcd7641d0bd905a59"
server_key_iv: "d425d8c427e4c70b7fa665a3d837f475"
server_key_version: "encryptionkey@0"
I20260925 14:52:30.049856 21210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "21f69003df1f46d69b32f3dea449d7bf"
format_stamp: "Formatted at 2026-09-25 14:52:30 on dist-test-slave-v5vl"
server_key: "08e9249af40be4ddcd7641d0bd905a59"
server_key_iv: "d425d8c427e4c70b7fa665a3d837f475"
server_key_version: "encryptionkey@0"
I20260925 14:52:30.054563 21210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:52:30.057708 21225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:30.058941 21210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:30.059087 21210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "21f69003df1f46d69b32f3dea449d7bf"
format_stamp: "Formatted at 2026-09-25 14:52:30 on dist-test-slave-v5vl"
server_key: "08e9249af40be4ddcd7641d0bd905a59"
server_key_iv: "d425d8c427e4c70b7fa665a3d837f475"
server_key_version: "encryptionkey@0"
I20260925 14:52:30.059278 21210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:30.073518 21210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:30.078166 21210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:30.078456 21210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:30.079450 21210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:30.080905 21210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:30.080988 21210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:30.081091 21210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:30.081152 21210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:30.113911 21210 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:37829
I20260925 14:52:30.113960 21337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:37829 every 8 connection(s)
I20260925 14:52:30.115222 21210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:30.117552 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 21210
I20260925 14:52:30.117784 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:30.118271 17943 external_mini_cluster.cc:1428] Setting key 22c30eb0de21cef7e75c6bfa97ba7073
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347950, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:30.136262 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:30.117849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:39677 (local address 127.17.133.254:33725)
0925 14:52:30.118040 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:30.118047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:30.119248 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 14:52:30.120950 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:30.120958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:30.121040 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:30.121141 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:30.122530 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.123711 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.125303 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.125704 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.125837 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:30.129404 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:30.129414 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:30.129422 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:30.129467 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:30.133077 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.133788 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.133796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.133802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.133969 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.134398 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.134405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.134412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.134800 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:30.134948 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:30.135608 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 14:52:30.135849 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
I20260925 14:52:30.137125 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:30.118341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33725 (local address 127.17.133.194:39677)
0925 14:52:30.119124 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:30.119164 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:30.120709 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:30.121317 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:30.121324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:30.121828 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:30.122352 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:30.122367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.123848 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.123851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:30.125142 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:30.125153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.125945 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.125952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:30.126173 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:30.126860 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:30.126881 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:30.129179 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:30.133203 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:30.133210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:30.133229 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:30.133651 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:30.134092 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:30.134095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:30.134097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:30.134273 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:30.134916 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:30.134922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:30.135235 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 14:52:30.135444 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 14:52:30.135809 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":341,"threads_started":1}
I20260925 14:52:30.138599 21338 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33725
I20260925 14:52:30.138949 21338 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
I20260925 14:52:30.139763 21338 heartbeater.cc:511] Master 127.17.133.254:33725 requested a full tablet report, sending...
I20260925 14:52:30.141397 21015 ts_manager.cc:194] Registered new tserver with Master: 21f69003df1f46d69b32f3dea449d7bf (127.17.133.194:37829)
I20260925 14:52:30.142211 21015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:39677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:52:30.161885 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33725
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:30.492970 21343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:30.493285 21343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:30.493455 21343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:30.500749 21343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:30.500911 21343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:30.501083 21343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:30.511549 21343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:30.512979 21343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:30.514242 21343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:30.526988 21348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:30.527014 21351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:30.526937 21349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:30.527818 21343 server_base.cc:1034] running on GCE node
I20260925 14:52:30.528443 21343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:30.529471 21343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:30.530700 21343 hybrid_clock.cc:648] HybridClock initialized: now 1790347950530656 us; error 56 us; skew 500 ppm
I20260925 14:52:30.531013 21343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347950, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:30.534065 21343 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:52:30.535444 21343 webserver.cc:460] Webserver started at http://127.17.133.195:46355/ using document root <none> and password file <none>
I20260925 14:52:30.536120 21343 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:30.536259 21343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:30.536553 21343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:30.539358 21343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "45e00d81d74f4d05946566003f646ff8"
format_stamp: "Formatted at 2026-09-25 14:52:30 on dist-test-slave-v5vl"
server_key: "05c4aabaf4f655502da13c604b5bacb1"
server_key_iv: "00a35e9a23a0dad6bd86301b928c3f52"
server_key_version: "encryptionkey@0"
I20260925 14:52:30.540164 21343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "45e00d81d74f4d05946566003f646ff8"
format_stamp: "Formatted at 2026-09-25 14:52:30 on dist-test-slave-v5vl"
server_key: "05c4aabaf4f655502da13c604b5bacb1"
server_key_iv: "00a35e9a23a0dad6bd86301b928c3f52"
server_key_version: "encryptionkey@0"
I20260925 14:52:30.545307 21343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 14:52:30.548521 21358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:30.549877 21343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:52:30.550051 21343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "45e00d81d74f4d05946566003f646ff8"
format_stamp: "Formatted at 2026-09-25 14:52:30 on dist-test-slave-v5vl"
server_key: "05c4aabaf4f655502da13c604b5bacb1"
server_key_iv: "00a35e9a23a0dad6bd86301b928c3f52"
server_key_version: "encryptionkey@0"
I20260925 14:52:30.550237 21343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:30.562073 21343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:30.565681 21343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:30.565980 21343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:30.566802 21343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:30.568008 21343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:30.568078 21343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:30.568248 21343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:30.568297 21343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:30.598755 21343 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:39109
I20260925 14:52:30.598798 21470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:39109 every 8 connection(s)
I20260925 14:52:30.599988 21343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:30.600662 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 21343
I20260925 14:52:30.600901 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:30.601599 17943 external_mini_cluster.cc:1428] Setting key 2fee8090dedc7f7a078b164a6171869b
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347950, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:30.618777 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:30.603143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:35049 (local address 127.17.133.254:33725)
0925 14:52:30.603639 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:30.603648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:30.604300 (+ 652us) server_negotiation.cc:407] Connection header received
0925 14:52:30.606332 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:30.606337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:30.606410 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:30.606486 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:30.607817 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.608496 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.610144 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.610556 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.610643 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:30.613777 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:30.613782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:30.613785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:30.613814 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:30.616295 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.617136 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.617140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.617142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.617246 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.617585 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.617588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.617590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.617816 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:30.617913 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:30.618427 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 14:52:30.618563 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260925 14:52:30.619612 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:30.603210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33725 (local address 127.17.133.195:35049)
0925 14:52:30.604125 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:30.604174 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:30.606091 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:30.606685 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:30.606693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:30.607156 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:30.607687 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:30.607704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.608663 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.608666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:30.610024 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:30.610040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.610726 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.610731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:30.610910 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:30.611557 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:30.611578 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:30.613597 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:30.616425 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:30.616433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:30.616449 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:30.616992 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:30.617334 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:30.617337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:30.617339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:30.617498 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:30.617906 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:30.617910 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:30.618156 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 14:52:30.618351 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 14:52:30.618669 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":342,"threads_started":1}
I20260925 14:52:30.620754 21471 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33725
I20260925 14:52:30.620995 21471 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:30.621613 21471 heartbeater.cc:511] Master 127.17.133.254:33725 requested a full tablet report, sending...
I20260925 14:52:30.622742 21015 ts_manager.cc:194] Registered new tserver with Master: 45e00d81d74f4d05946566003f646ff8 (127.17.133.195:39109)
I20260925 14:52:30.623447 21015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:35049
I20260925 14:52:30.628772 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:30.673491 21204 heartbeater.cc:503] Master 127.17.133.254:33725 was elected leader, sending a full tablet report...
I20260925 14:52:30.674986 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:30.661727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.17.133.254:33725)
0925 14:52:30.661908 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:30.661912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:30.662399 (+ 487us) server_negotiation.cc:407] Connection header received
0925 14:52:30.662540 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:30.662562 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:30.662650 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:30.662739 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:30.664002 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.664749 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.665986 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.666385 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.666518 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:30.669927 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:30.669932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:30.669935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:30.669965 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:30.672480 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.673081 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.673085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.673088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.673229 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.673671 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.673675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.673678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.674036 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:30.674254 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:30.674555 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 14:52:30.674703 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 14:52:30.675565 21015 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58206
I20260925 14:52:30.676844 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21076
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:30.691757 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:30.677752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.17.133.193:46693)
0925 14:52:30.678350 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:30.678361 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:30.678389 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:52:30.678454 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:30.678462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:30.678660 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:30.678877 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:30.679997 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.681064 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.682209 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.682764 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.682898 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:30.686740 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:30.686762 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:30.686775 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:30.686807 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:30.689202 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.689866 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.689872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.689874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.689997 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.690356 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.690362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.690365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.690695 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:30.690975 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:30.691207 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 14:52:30.691447 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":304,"threads_started":1}
W20260925 14:52:30.692986 21199 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48324
W20260925 14:52:30.694067 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21076: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:52:30.694227 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 21076
I20260925 14:52:30.709949 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21210
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:52:30.725611 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:30.710821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.17.133.194:37829)
0925 14:52:30.711298 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:30.711305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:30.711328 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:52:30.711387 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:30.711393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:30.711602 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:30.711770 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:30.713079 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.714041 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.715274 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.715818 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.715955 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:30.720092 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:30.720101 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:30.720116 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:30.720154 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:30.722860 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.723477 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.723486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.723488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.723637 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.724009 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.724016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.724019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.724343 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:30.724672 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:30.725008 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 14:52:30.725267 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":239,"threads_started":1}
W20260925 14:52:30.726779 21333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46150
W20260925 14:52:30.727819 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:52:30.727995 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 21210
I20260925 14:52:30.743252 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21343
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:52:30.759573 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:30.744187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.17.133.195:39109)
0925 14:52:30.744840 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:30.744848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:30.744876 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:52:30.744934 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:30.744940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:30.745166 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:30.745336 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:30.746601 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.747621 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.748963 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:30.749541 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:30.749683 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:30.754075 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:30.754091 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:30.754111 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:30.754153 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:30.756870 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.757502 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.757517 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.757519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.757629 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:30.757986 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:30.758000 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:30.758003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:30.758329 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:30.758638 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:30.758951 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 14:52:30.759226 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":306,"threads_started":1}
W20260925 14:52:30.761023 21466 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54230
W20260925 14:52:30.762102 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21343: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:52:30.762251 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 21343
I20260925 14:52:30.777549 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 20983
I20260925 14:52:30.778632 21045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:30.878160 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 20983
2026-09-25T14:52:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3484 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[21488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[21488](info): set up 2 sockets
Sep 25 14:52:30 dist-test-slave-v5vl krb5kdc[21488](info): commencing operation
krb5kdc: starting...
W20260925 14:52:31.941229 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:32 dist-test-slave-v5vl krb5kdc[21488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:52:32.124349 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46715
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:32.441946 21501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:32.442250 21501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:32.442435 21501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:32.449846 21501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:32.449970 21501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:32.450045 21501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:32.450145 21501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:32.450220 21501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:32.459903 21501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:46715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:32.461131 21501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:32.462313 21501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:32.473362 21506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:32.473322 21507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:32.473779 21509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:32.474375 21501 server_base.cc:1034] running on GCE node
I20260925 14:52:32.475100 21501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:32.476408 21501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:32.477633 21501 hybrid_clock.cc:648] HybridClock initialized: now 1790347952477601 us; error 42 us; skew 500 ppm
I20260925 14:52:32.477983 21501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:32 dist-test-slave-v5vl krb5kdc[21488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347952, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:32.484370 21501 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:52:32.485913 21501 webserver.cc:460] Webserver started at http://127.17.133.254:46003/ using document root <none> and password file <none>
I20260925 14:52:32.486585 21501 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:32.486724 21501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:32.487043 21501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:32.489951 21501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "6da2ea734b114dfbb5931e4fdb03e8a4"
format_stamp: "Formatted at 2026-09-25 14:52:32 on dist-test-slave-v5vl"
server_key: "9399f97b2c3f752b5eaf9918c8876499"
server_key_iv: "5a35376d7a5ad68053e17525d44498e9"
server_key_version: "encryptionkey@0"
I20260925 14:52:32.490770 21501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "6da2ea734b114dfbb5931e4fdb03e8a4"
format_stamp: "Formatted at 2026-09-25 14:52:32 on dist-test-slave-v5vl"
server_key: "9399f97b2c3f752b5eaf9918c8876499"
server_key_iv: "5a35376d7a5ad68053e17525d44498e9"
server_key_version: "encryptionkey@0"
I20260925 14:52:32.495678 21501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 14:52:32.499040 21516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:32.500233 21501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:52:32.500415 21501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "6da2ea734b114dfbb5931e4fdb03e8a4"
format_stamp: "Formatted at 2026-09-25 14:52:32 on dist-test-slave-v5vl"
server_key: "9399f97b2c3f752b5eaf9918c8876499"
server_key_iv: "5a35376d7a5ad68053e17525d44498e9"
server_key_version: "encryptionkey@0"
I20260925 14:52:32.500615 21501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:32.523342 21501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:32.527280 21501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:32.527572 21501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:32.545923 21501 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:46715
I20260925 14:52:32.545976 21567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:46715 every 8 connection(s)
I20260925 14:52:32.547351 21501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:32.550873 21568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:32.557261 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 21501
I20260925 14:52:32.557464 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:32.558100 17943 external_mini_cluster.cc:1428] Setting key b9b3d35106155f017485b332e2ad4eb3
I20260925 14:52:32.561146 21568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4: Bootstrap starting.
I20260925 14:52:32.564494 21568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:32.565670 21568 log.cc:824] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:32.568849 21568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4: No bootstrap required, opened a new log
Sep 25 14:52:32 dist-test-slave-v5vl krb5kdc[21488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:32.577386 21568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da2ea734b114dfbb5931e4fdb03e8a4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46715 } }
I20260925 14:52:32.577744 21568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:32.577910 21568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da2ea734b114dfbb5931e4fdb03e8a4, State: Initialized, Role: FOLLOWER
I20260925 14:52:32.578616 21568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da2ea734b114dfbb5931e4fdb03e8a4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46715 } }
I20260925 14:52:32.578874 21568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:32.579027 21568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:32.579259 21568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:32.581116 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:32.560012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.17.133.254:46715)
0925 14:52:32.560791 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:32.560802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:32.560844 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:52:32.562552 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:32.562574 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:32.562952 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:32.563267 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:32.565160 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:32.566858 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:32.568627 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:32.569367 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:32.569489 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:32.573971 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:32.573980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:32.573998 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:32.574031 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:32.577142 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:32.577827 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:32.577844 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:32.577850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:32.578011 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:32.578436 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:32.578440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:32.578442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:32.578956 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:32.579243 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:32.579661 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 14:52:32.580037 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":332,"threads_started":1}
I20260925 14:52:32.581964 21568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da2ea734b114dfbb5931e4fdb03e8a4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46715 } }
I20260925 14:52:32.582525 21568 leader_election.cc:302] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6da2ea734b114dfbb5931e4fdb03e8a4; no voters:
I20260925 14:52:32.583068 21568 leader_election.cc:288] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:32.583257 21573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:32.584226 21573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [term 1 LEADER]: Becoming Leader. State: Replica: 6da2ea734b114dfbb5931e4fdb03e8a4, State: Running, Role: LEADER
I20260925 14:52:32.584833 21573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da2ea734b114dfbb5931e4fdb03e8a4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46715 } }
I20260925 14:52:32.585129 21568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:32.587546 21574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6da2ea734b114dfbb5931e4fdb03e8a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da2ea734b114dfbb5931e4fdb03e8a4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46715 } } }
I20260925 14:52:32.587622 21575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6da2ea734b114dfbb5931e4fdb03e8a4. Latest consensus state: current_term: 1 leader_uuid: "6da2ea734b114dfbb5931e4fdb03e8a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da2ea734b114dfbb5931e4fdb03e8a4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46715 } } }
I20260925 14:52:32.587919 21574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:32.588021 21575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4 [sys.catalog]: This master's current role is: LEADER
W20260925 14:52:32.595687 21588 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:52:32.595796 21588 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:52:32.595963 21582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:32.597857 21582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:32.605198 21582 catalog_manager.cc:1347] Generated new cluster ID: 0df27ecc02094414a7b714fed5bdf352
I20260925 14:52:32.605298 21582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:32.637061 21582 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:32.638408 21582 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:32.652099 21582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6da2ea734b114dfbb5931e4fdb03e8a4: Generated new TSK 0
I20260925 14:52:32.653023 21582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:52:32.708163 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46715
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:33.053803 21594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:33.054095 21594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:33.054265 21594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:33.061504 21594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:33.061610 21594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:33.061787 21594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:33.071452 21594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:33.072762 21594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:33.074038 21594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:33.087268 21599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:33.087314 21602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:33.087472 21600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:33.088445 21594 server_base.cc:1034] running on GCE node
I20260925 14:52:33.089076 21594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:33.089959 21594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:33.091161 21594 hybrid_clock.cc:648] HybridClock initialized: now 1790347953091125 us; error 34 us; skew 500 ppm
I20260925 14:52:33.091480 21594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:33 dist-test-slave-v5vl krb5kdc[21488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347953, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:33.094691 21594 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:52:33.096127 21594 webserver.cc:460] Webserver started at http://127.17.133.193:44151/ using document root <none> and password file <none>
I20260925 14:52:33.096912 21594 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:33.097045 21594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:33.097366 21594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:33.100232 21594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "cb8880ff8d4f4542902b203ada60f092"
format_stamp: "Formatted at 2026-09-25 14:52:33 on dist-test-slave-v5vl"
server_key: "58b9ac9a6ce95f5508a1308af1462293"
server_key_iv: "109c6568914397709d74024edaf417b9"
server_key_version: "encryptionkey@0"
I20260925 14:52:33.101055 21594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "cb8880ff8d4f4542902b203ada60f092"
format_stamp: "Formatted at 2026-09-25 14:52:33 on dist-test-slave-v5vl"
server_key: "58b9ac9a6ce95f5508a1308af1462293"
server_key_iv: "109c6568914397709d74024edaf417b9"
server_key_version: "encryptionkey@0"
I20260925 14:52:33.106169 21594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:52:33.109463 21609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:33.110743 21594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:52:33.110896 21594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "cb8880ff8d4f4542902b203ada60f092"
format_stamp: "Formatted at 2026-09-25 14:52:33 on dist-test-slave-v5vl"
server_key: "58b9ac9a6ce95f5508a1308af1462293"
server_key_iv: "109c6568914397709d74024edaf417b9"
server_key_version: "encryptionkey@0"
I20260925 14:52:33.111090 21594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:33.135939 21594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:33.139992 21594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:33.140290 21594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:33.141146 21594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:33.142383 21594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:33.142454 21594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:33.142606 21594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:33.142695 21594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:33.175112 21594 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:42253
I20260925 14:52:33.175154 21721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:42253 every 8 connection(s)
I20260925 14:52:33.176366 21594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:33.178459 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 21594
I20260925 14:52:33.178663 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:33.179545 17943 external_mini_cluster.cc:1428] Setting key 729386b046c3757f228b1aa0db6c08b9
Sep 25 14:52:33 dist-test-slave-v5vl krb5kdc[21488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347953, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:33.199400 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:33.179047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:58809 (local address 127.17.133.254:46715)
0925 14:52:33.179530 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:33.179537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:33.180576 (+ 1039us) server_negotiation.cc:407] Connection header received
0925 14:52:33.183019 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:33.183023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:33.183081 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:33.183161 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:33.185266 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:33.186067 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:33.188225 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:33.188997 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:33.189134 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:33.192631 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:33.192640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:33.192648 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:33.192692 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:33.196304 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:33.197052 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:33.197061 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:33.197066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:33.197230 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:33.197628 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:33.197634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:33.197637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:33.197988 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:33.198126 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:33.198767 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 14:52:33.198987 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":248,"threads_started":1}
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:52:33.200079 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:33.179608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:46715 (local address 127.17.133.193:58809)
0925 14:52:33.180409 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:33.180455 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:33.182701 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:33.183411 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:33.183424 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:33.184222 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:33.185055 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:33.185085 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:33.186229 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:33.186235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:33.188072 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:33.188086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:33.189225 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:33.189231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:33.189438 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:33.190131 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:33.190152 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:33.192406 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:33.196440 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:33.196447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:33.196460 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:33.196936 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:33.197338 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:33.197341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:33.197343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:33.197515 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:33.198106 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:33.198112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:33.198403 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 14:52:33.198610 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 14:52:33.198977 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":282,"threads_started":1}
I20260925 14:52:33.201606 21722 heartbeater.cc:347] Connected to a master server at 127.17.133.254:46715
I20260925 14:52:33.201853 21722 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
I20260925 14:52:33.202582 21722 heartbeater.cc:511] Master 127.17.133.254:46715 requested a full tablet report, sending...
I20260925 14:52:33.204785 21533 ts_manager.cc:194] Registered new tserver with Master: cb8880ff8d4f4542902b203ada60f092 (127.17.133.193:42253)
I20260925 14:52:33.206434 21533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:58809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:52:33.223867 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46715
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:33.537735 21728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:33.538015 21728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:33.538127 21728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:33.545310 21728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:33.545456 21728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:33.545650 21728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:33.555387 21728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:33.556751 21728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:33.558105 21728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:33.570897 21733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:33.570941 21736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:33.571033 21734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:33.572018 21728 server_base.cc:1034] running on GCE node
I20260925 14:52:33.572657 21728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:33.573498 21728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:33.574710 21728 hybrid_clock.cc:648] HybridClock initialized: now 1790347953574676 us; error 45 us; skew 500 ppm
I20260925 14:52:33.575026 21728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:33 dist-test-slave-v5vl krb5kdc[21488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347953, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:33.578018 21728 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:52:33.579432 21728 webserver.cc:460] Webserver started at http://127.17.133.194:42507/ using document root <none> and password file <none>
I20260925 14:52:33.580101 21728 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:33.580241 21728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:33.580540 21728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:33.583293 21728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "4a3f3c818b084f3a961d9bd623e9683c"
format_stamp: "Formatted at 2026-09-25 14:52:33 on dist-test-slave-v5vl"
server_key: "c2f487557ff8c5a20fb596f55e56a970"
server_key_iv: "9fcf19afe7f020bcf24dfaa603245dd2"
server_key_version: "encryptionkey@0"
I20260925 14:52:33.584028 21728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "4a3f3c818b084f3a961d9bd623e9683c"
format_stamp: "Formatted at 2026-09-25 14:52:33 on dist-test-slave-v5vl"
server_key: "c2f487557ff8c5a20fb596f55e56a970"
server_key_iv: "9fcf19afe7f020bcf24dfaa603245dd2"
server_key_version: "encryptionkey@0"
I20260925 14:52:33.589121 21728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 14:52:33.592324 21743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:33.593559 21728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:52:33.593722 21728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "4a3f3c818b084f3a961d9bd623e9683c"
format_stamp: "Formatted at 2026-09-25 14:52:33 on dist-test-slave-v5vl"
server_key: "c2f487557ff8c5a20fb596f55e56a970"
server_key_iv: "9fcf19afe7f020bcf24dfaa603245dd2"
server_key_version: "encryptionkey@0"
I20260925 14:52:33.593940 21728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:33.616117 21728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:33.619810 21728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:33.620090 21728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:33.621028 21728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:33.622195 21728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:33.622290 21728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:33.622421 21728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:33.622503 21728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:33.654047 21728 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:40735
I20260925 14:52:33.654152 21855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:40735 every 8 connection(s)
I20260925 14:52:33.655308 21728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:33.662048 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 21728
I20260925 14:52:33.662206 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:33.662768 17943 external_mini_cluster.cc:1428] Setting key e8dead7f55d2ef88259fbcdf747c835a
Sep 25 14:52:33 dist-test-slave-v5vl krb5kdc[21488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347953, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:33.675660 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:33.657755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:34181 (local address 127.17.133.254:46715)
0925 14:52:33.657950 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:33.657956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:33.659108 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 14:52:33.660862 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:33.660867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:33.660940 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:33.661025 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:33.662759 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:33.663505 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:33.665448 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:33.665843 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:33.665945 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:33.669971 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:33.669980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:33.669988 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:33.670037 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:33.672649 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:33.673530 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:33.673534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:33.673536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:33.673636 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:33.674077 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:33.674080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:33.674082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:33.674332 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:33.674440 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:33.675261 (+ 821us) server_negotiation.cc:299] Negotiation successful
0925 14:52:33.675432 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:52:33.677208 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:33.658295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:46715 (local address 127.17.133.194:34181)
0925 14:52:33.658954 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:33.658996 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:33.660570 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:33.661226 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:33.661234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:33.661799 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:33.662595 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:33.662628 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:33.663636 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:33.663642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:33.665315 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:33.665330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:33.666066 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:33.666075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:33.666399 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:33.667107 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:33.667139 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:33.669723 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:33.672766 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:33.672777 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:33.672795 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:33.673423 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:33.673730 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:33.673737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:33.673744 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:33.673976 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:33.674449 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:33.674460 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:33.674887 (+ 427us) client_negotiation.cc:769] Sending connection context
0925 14:52:33.675189 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 14:52:33.675745 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":256,"threads_started":1}
I20260925 14:52:33.678649 21856 heartbeater.cc:347] Connected to a master server at 127.17.133.254:46715
I20260925 14:52:33.678937 21856 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:33.679590 21856 heartbeater.cc:511] Master 127.17.133.254:46715 requested a full tablet report, sending...
I20260925 14:52:33.681025 21533 ts_manager.cc:194] Registered new tserver with Master: 4a3f3c818b084f3a961d9bd623e9683c (127.17.133.194:40735)
I20260925 14:52:33.681788 21533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:34181
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:52:33.705466 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46715
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:34.015374 21861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:34.015666 21861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:34.015817 21861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:34.023325 21861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:34.023433 21861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:34.023563 21861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:34.033296 21861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:34.034511 21861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:34.035748 21861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:34.048151 21866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:34.048106 21869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:34.048192 21867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:34.049934 21861 server_base.cc:1034] running on GCE node
I20260925 14:52:34.050416 21861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:34.051283 21861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:34.052496 21861 hybrid_clock.cc:648] HybridClock initialized: now 1790347954052459 us; error 50 us; skew 500 ppm
I20260925 14:52:34.052817 21861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[21488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347954, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:34.055713 21861 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:52:34.057227 21861 webserver.cc:460] Webserver started at http://127.17.133.195:37133/ using document root <none> and password file <none>
I20260925 14:52:34.057914 21861 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:34.058043 21861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:34.058360 21861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:34.061249 21861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "104b6a57ac814d1f829200d330f6279b"
format_stamp: "Formatted at 2026-09-25 14:52:34 on dist-test-slave-v5vl"
server_key: "97aeaf2d5f1e1dd96ee9ea57ba0c804a"
server_key_iv: "02d80be3118b652f81839c064f16ab65"
server_key_version: "encryptionkey@0"
I20260925 14:52:34.062026 21861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "104b6a57ac814d1f829200d330f6279b"
format_stamp: "Formatted at 2026-09-25 14:52:34 on dist-test-slave-v5vl"
server_key: "97aeaf2d5f1e1dd96ee9ea57ba0c804a"
server_key_iv: "02d80be3118b652f81839c064f16ab65"
server_key_version: "encryptionkey@0"
I20260925 14:52:34.067060 21861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:52:34.070513 21876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:34.071889 21861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:52:34.072067 21861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "104b6a57ac814d1f829200d330f6279b"
format_stamp: "Formatted at 2026-09-25 14:52:34 on dist-test-slave-v5vl"
server_key: "97aeaf2d5f1e1dd96ee9ea57ba0c804a"
server_key_iv: "02d80be3118b652f81839c064f16ab65"
server_key_version: "encryptionkey@0"
I20260925 14:52:34.072278 21861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:34.094312 21861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:34.100222 21861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:34.100541 21861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:34.101820 21861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:34.103438 21861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:34.103523 21861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:34.103597 21861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:34.103636 21861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:34.135528 21861 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:46739
I20260925 14:52:34.135589 21988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:46739 every 8 connection(s)
I20260925 14:52:34.136900 21861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:34.142949 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 21861
I20260925 14:52:34.143172 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:34.143620 17943 external_mini_cluster.cc:1428] Setting key bd848507753437f344c3c07d9026aa60
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[21488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347954, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:34.154697 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:34.139344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:40055 (local address 127.17.133.254:46715)
0925 14:52:34.139517 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:34.139522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:34.140550 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 14:52:34.142299 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:34.142304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:34.142412 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:34.142518 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:34.144046 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.144878 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.146248 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.146631 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.146748 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:34.149892 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:34.149896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:34.149899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:34.149928 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:34.152232 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.152950 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.152954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.152956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.153070 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.153428 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.153431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.153433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.153660 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:34.153770 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:34.154330 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 14:52:34.154468 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:52:34.155546 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:34.139762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:46715 (local address 127.17.133.195:40055)
0925 14:52:34.140399 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:34.140434 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:34.142046 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:34.142744 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:34.142751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:34.143322 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:34.143903 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:34.143920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.145015 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.145018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:34.146114 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:34.146123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.146840 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.146846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:34.147076 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:34.147647 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:34.147670 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:34.149625 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:34.152368 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:34.152374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:34.152383 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:34.152849 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:34.153156 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:34.153158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:34.153160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:34.153338 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:34.153763 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:34.153768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:34.154029 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 14:52:34.154237 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 14:52:34.154545 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":247,"threads_started":1}
I20260925 14:52:34.157014 21989 heartbeater.cc:347] Connected to a master server at 127.17.133.254:46715
I20260925 14:52:34.157284 21989 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:34.157997 21989 heartbeater.cc:511] Master 127.17.133.254:46715 requested a full tablet report, sending...
I20260925 14:52:34.159279 21533 ts_manager.cc:194] Registered new tserver with Master: 104b6a57ac814d1f829200d330f6279b (127.17.133.195:46739)
I20260925 14:52:34.160017 21533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:40055
I20260925 14:52:34.170647 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[21488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260925 14:52:34.209182 21722 heartbeater.cc:503] Master 127.17.133.254:46715 was elected leader, sending a full tablet report...
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[21488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:34.218281 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:34.203329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.17.133.193:42253)
0925 14:52:34.203963 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:34.203970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:34.203993 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:52:34.204056 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:34.204063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:34.204275 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:34.204481 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:34.205809 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.206806 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.208006 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.208569 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.208762 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:34.212681 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:34.212691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:34.212706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:34.212733 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:34.215532 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.216196 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.216208 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.216210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.216324 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.216783 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.216789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.216793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.217113 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:34.217422 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:34.217689 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 14:52:34.217938 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":330,"threads_started":1}
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[21488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:34.234442 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:34.222004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.17.133.254:46715)
0925 14:52:34.222204 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:34.222210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:34.222298 (+ 88us) server_negotiation.cc:407] Connection header received
0925 14:52:34.222438 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:34.222442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:34.222495 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:34.222567 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:34.223655 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.224300 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.225471 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.225847 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.225945 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:34.229517 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:34.229522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:34.229525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:34.229553 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:34.232058 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.232722 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.232726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.232729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.232851 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.233198 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.233202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.233204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.233490 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:34.233672 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:34.233928 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 14:52:34.234078 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 14:52:34.234987 21533 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39780
I20260925 14:52:34.235982 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21594
I20260925 14:52:34.248319 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:34.236936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.17.133.193:42253)
0925 14:52:34.237135 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:34.237141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:34.237327 (+ 186us) server_negotiation.cc:407] Connection header received
0925 14:52:34.237516 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:34.237521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:34.237612 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:34.237720 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:34.238862 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.239541 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.240849 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.241266 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.241408 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:34.243502 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:34.243507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:34.243510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:34.243540 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:34.245893 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.246497 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.246504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.246507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.246629 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.247030 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.247035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.247038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.247284 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:34.247515 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:34.247795 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:52:34.248005 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:52:34.248974 21717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:34.330145 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 21594
I20260925 14:52:34.345821 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21728
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[21488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:52:34.361202 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:34.346686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.17.133.194:40735)
0925 14:52:34.347212 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:34.347219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:34.347262 (+ 43us) server_negotiation.cc:407] Connection header received
0925 14:52:34.347321 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:34.347327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:34.347539 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:34.347844 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:34.349027 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.349991 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.351261 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.351796 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.351896 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:34.355912 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:34.355924 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:34.355940 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:34.355980 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:34.358715 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.359397 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.359404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.359406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.359525 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.359829 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.359833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.359834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.360140 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:34.360374 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:34.360733 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 14:52:34.360923 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":293,"threads_started":1}
I20260925 14:52:34.362450 21851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:34.442895 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 21728
I20260925 14:52:34.458156 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21861
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[21488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:52:34.473117 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:34.459079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.17.133.195:46739)
0925 14:52:34.459695 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:34.459702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:34.459729 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:52:34.459785 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:34.459791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:34.459994 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:34.460122 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:34.461357 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.462357 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.463568 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:34.464083 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:34.464174 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:34.468158 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:34.468168 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:34.468181 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:34.468213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:34.470715 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.471238 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.471244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.471246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.471395 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:34.471705 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:34.471710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:34.471713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:34.472064 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:34.472279 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:34.472483 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 14:52:34.472729 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":325,"threads_started":1}
I20260925 14:52:34.474330 21984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:34.561091 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 21861
I20260925 14:52:34.576206 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 21501
I20260925 14:52:34.577458 21563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:34.677922 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 21501
2026-09-25T14:52:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3799 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[22008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[22008](info): set up 2 sockets
Sep 25 14:52:34 dist-test-slave-v5vl krb5kdc[22008](info): commencing operation
krb5kdc: starting...
W20260925 14:52:35.757226 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:35 dist-test-slave-v5vl krb5kdc[22008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:52:35.945537 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45143
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:45143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:36.256116 22021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:36.256387 22021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:36.256562 22021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:36.263825 22021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:36.263952 22021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:36.264000 22021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:36.264050 22021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:36.264124 22021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:36.273818 22021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:45143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:36.275040 22021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:36.276298 22021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:36.287158 22027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.287220 22029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.287124 22026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:36.288081 22021 server_base.cc:1034] running on GCE node
I20260925 14:52:36.288978 22021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:36.290298 22021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:36.291522 22021 hybrid_clock.cc:648] HybridClock initialized: now 1790347956291486 us; error 48 us; skew 500 ppm
I20260925 14:52:36.291855 22021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:36 dist-test-slave-v5vl krb5kdc[22008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347956, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:36.294807 22021 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:52:36.295939 22021 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:36.296066 22021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:36.296377 22021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:36.299261 22021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "76335e76149842cba144ace969eeb9e6"
format_stamp: "Formatted at 2026-09-25 14:52:36 on dist-test-slave-v5vl"
server_key: "b14d0617eedbdcb5eaf155a2aa342e5c"
server_key_iv: "d2e76dfc21ed0def1e4a4c996371b7e6"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.300045 22021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "76335e76149842cba144ace969eeb9e6"
format_stamp: "Formatted at 2026-09-25 14:52:36 on dist-test-slave-v5vl"
server_key: "b14d0617eedbdcb5eaf155a2aa342e5c"
server_key_iv: "d2e76dfc21ed0def1e4a4c996371b7e6"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.304836 22021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.007s
I20260925 14:52:36.308192 22035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.309445 22021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:36.309623 22021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "76335e76149842cba144ace969eeb9e6"
format_stamp: "Formatted at 2026-09-25 14:52:36 on dist-test-slave-v5vl"
server_key: "b14d0617eedbdcb5eaf155a2aa342e5c"
server_key_iv: "d2e76dfc21ed0def1e4a4c996371b7e6"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.309835 22021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:36.332649 22021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:36.336577 22021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:36.336927 22021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:36.354803 22021 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:45143
I20260925 14:52:36.354794 22086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:45143 every 8 connection(s)
I20260925 14:52:36.355818 22021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:36.359445 22087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:36.364768 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 22021
I20260925 14:52:36.364979 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:36.365595 17943 external_mini_cluster.cc:1428] Setting key 9b672c3dc4f1f69fc0db7f88801e0476
I20260925 14:52:36.370754 22087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6: Bootstrap starting.
I20260925 14:52:36.374387 22087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:36.375573 22087 log.cc:824] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:36.378827 22087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6: No bootstrap required, opened a new log
Sep 25 14:52:36 dist-test-slave-v5vl krb5kdc[22008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:36.389390 22087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76335e76149842cba144ace969eeb9e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45143 } }
I20260925 14:52:36.389796 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.367494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.17.133.254:45143)
0925 14:52:36.368309 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:36.368321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:36.368364 (+ 43us) server_negotiation.cc:407] Connection header received
0925 14:52:36.370093 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:36.370115 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:36.370477 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:36.370771 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:36.372638 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:36.373912 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:36.375740 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:36.376778 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:36.376943 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:36.381354 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:36.381365 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:36.381388 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:36.381442 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:36.385314 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:36.385959 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:36.385968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:36.385977 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:36.386202 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:36.386527 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:36.386532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:36.386535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:36.387271 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:36.387573 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:36.387886 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 14:52:36.388385 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":388,"threads_started":1}
I20260925 14:52:36.389849 22087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:36.389935 22087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76335e76149842cba144ace969eeb9e6, State: Initialized, Role: FOLLOWER
I20260925 14:52:36.390601 22087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76335e76149842cba144ace969eeb9e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45143 } }
I20260925 14:52:36.390848 22087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:36.390995 22087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:36.391193 22087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:36.393594 22087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76335e76149842cba144ace969eeb9e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45143 } }
I20260925 14:52:36.394086 22087 leader_election.cc:302] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76335e76149842cba144ace969eeb9e6; no voters:
I20260925 14:52:36.394627 22087 leader_election.cc:288] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:36.394758 22092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:36.395462 22092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 76335e76149842cba144ace969eeb9e6, State: Running, Role: LEADER
I20260925 14:52:36.396008 22092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76335e76149842cba144ace969eeb9e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45143 } }
I20260925 14:52:36.396792 22087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:36.399273 22093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76335e76149842cba144ace969eeb9e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76335e76149842cba144ace969eeb9e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45143 } } }
I20260925 14:52:36.399631 22093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:36.403465 22094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76335e76149842cba144ace969eeb9e6. Latest consensus state: current_term: 1 leader_uuid: "76335e76149842cba144ace969eeb9e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76335e76149842cba144ace969eeb9e6" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45143 } } }
I20260925 14:52:36.403766 22094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:36.405076 22099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:36.410480 22099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:36.418824 22099 catalog_manager.cc:1347] Generated new cluster ID: d3eb92ab25f44f18bba2c0677f988885
I20260925 14:52:36.418952 22099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:36.433248 22099 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:36.434518 22099 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:36.445076 22099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6: Generated new TSK 0
I20260925 14:52:36.445874 22099 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:52:36.493280 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45143
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:36.808651 22113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:36.808956 22113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:36.809126 22113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:36.816426 22113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:36.816573 22113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:36.816766 22113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:36.826309 22113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:36.827579 22113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:36.828927 22113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:36.841662 22119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.841853 22118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.842101 22121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:36.842537 22113 server_base.cc:1034] running on GCE node
I20260925 14:52:36.843173 22113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:36.844105 22113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:36.845329 22113 hybrid_clock.cc:648] HybridClock initialized: now 1790347956845288 us; error 53 us; skew 500 ppm
I20260925 14:52:36.845641 22113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:36 dist-test-slave-v5vl krb5kdc[22008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347956, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:36.848666 22113 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:52:36.849794 22113 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:36.849931 22113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:36.850222 22113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:36.853032 22113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "dfacbd1b680b4b3d9c080e406a7c6ced"
format_stamp: "Formatted at 2026-09-25 14:52:36 on dist-test-slave-v5vl"
server_key: "6d96056b8f51b51c8edacb1af484ad2e"
server_key_iv: "389e7874c3eebe39125fbb693ece8c2d"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.853814 22113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "dfacbd1b680b4b3d9c080e406a7c6ced"
format_stamp: "Formatted at 2026-09-25 14:52:36 on dist-test-slave-v5vl"
server_key: "6d96056b8f51b51c8edacb1af484ad2e"
server_key_iv: "389e7874c3eebe39125fbb693ece8c2d"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.858517 22113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 14:52:36.861593 22127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.862737 22113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:52:36.862903 22113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "dfacbd1b680b4b3d9c080e406a7c6ced"
format_stamp: "Formatted at 2026-09-25 14:52:36 on dist-test-slave-v5vl"
server_key: "6d96056b8f51b51c8edacb1af484ad2e"
server_key_iv: "389e7874c3eebe39125fbb693ece8c2d"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.863103 22113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:36.887639 22113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:36.891654 22113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:36.891923 22113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:36.892855 22113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:36.894114 22113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:36.894228 22113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.894356 22113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:36.894438 22113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.926015 22113 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:45335
I20260925 14:52:36.926097 22239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:45335 every 8 connection(s)
I20260925 14:52:36.926826 22113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:36.933430 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 22113
I20260925 14:52:36.933630 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:36.934178 17943 external_mini_cluster.cc:1428] Setting key 47bc2f41a57b9f36a4f0e130deae8704
Sep 25 14:52:36 dist-test-slave-v5vl krb5kdc[22008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347956, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:36.947125 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.929103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:50235 (local address 127.17.133.254:45143)
0925 14:52:36.929503 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:36.929507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:36.930320 (+ 813us) server_negotiation.cc:407] Connection header received
0925 14:52:36.932146 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:36.932152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:36.932233 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:36.932345 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:36.933929 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:36.934598 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:36.936069 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:36.936437 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:36.936525 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:36.940787 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:36.940791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:36.940794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:36.940822 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:36.944053 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:36.944843 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:36.944851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:36.944854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:36.944975 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:36.945400 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:36.945405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:36.945410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:36.945674 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:36.945827 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:36.946508 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 14:52:36.946724 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":216,"threads_started":1}
I20260925 14:52:36.947974 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.929498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45143 (local address 127.17.133.193:50235)
0925 14:52:36.930144 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.930184 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:36.931883 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:36.932655 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:36.932664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:36.933149 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:36.933778 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:36.933806 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:36.934758 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:36.934762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:36.935929 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:36.935942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:36.936680 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:36.936689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:36.937007 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:36.937724 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:36.937758 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:36.940573 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:36.944199 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:36.944218 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:36.944236 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:36.944716 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:36.945086 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:36.945092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:36.945095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:36.945270 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:36.945802 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:36.945810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:36.946143 (+ 333us) client_negotiation.cc:769] Sending connection context
0925 14:52:36.946424 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 14:52:36.946910 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":285,"threads_started":1}
I20260925 14:52:36.949471 22240 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45143
I20260925 14:52:36.949805 22240 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:36.950598 22240 heartbeater.cc:511] Master 127.17.133.254:45143 requested a full tablet report, sending...
I20260925 14:52:36.952798 22052 ts_manager.cc:194] Registered new tserver with Master: dfacbd1b680b4b3d9c080e406a7c6ced (127.17.133.193:45335)
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:52:36.954252 22052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:50235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:52:36.979089 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45143
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:37.284482 22246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:37.284799 22246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:37.284988 22246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:37.291657 22246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:37.291797 22246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:37.291980 22246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:37.301232 22246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:37.302521 22246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:37.303767 22246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:37.316233 22251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:37.316314 22252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:37.316279 22254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:37.317493 22246 server_base.cc:1034] running on GCE node
I20260925 14:52:37.318053 22246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:37.318964 22246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:37.320150 22246 hybrid_clock.cc:648] HybridClock initialized: now 1790347957320094 us; error 53 us; skew 500 ppm
I20260925 14:52:37.320451 22246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:37 dist-test-slave-v5vl krb5kdc[22008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347957, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:37.323386 22246 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:52:37.324515 22246 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:37.324687 22246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:37.325007 22246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:37.327841 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "4dbd17896d6144b0b7ae76bdc86948ae"
format_stamp: "Formatted at 2026-09-25 14:52:37 on dist-test-slave-v5vl"
server_key: "b6a5935da230edae85044afd7aeaeb47"
server_key_iv: "76c99cd854cf2e6c9d2085c515d6c26a"
server_key_version: "encryptionkey@0"
I20260925 14:52:37.328658 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "4dbd17896d6144b0b7ae76bdc86948ae"
format_stamp: "Formatted at 2026-09-25 14:52:37 on dist-test-slave-v5vl"
server_key: "b6a5935da230edae85044afd7aeaeb47"
server_key_iv: "76c99cd854cf2e6c9d2085c515d6c26a"
server_key_version: "encryptionkey@0"
I20260925 14:52:37.333734 22246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 14:52:37.337044 22260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:37.338315 22246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 14:52:37.338478 22246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "4dbd17896d6144b0b7ae76bdc86948ae"
format_stamp: "Formatted at 2026-09-25 14:52:37 on dist-test-slave-v5vl"
server_key: "b6a5935da230edae85044afd7aeaeb47"
server_key_iv: "76c99cd854cf2e6c9d2085c515d6c26a"
server_key_version: "encryptionkey@0"
I20260925 14:52:37.338696 22246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:37.353708 22246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:37.358521 22246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:37.358796 22246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:37.359792 22246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:37.361359 22246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:37.361459 22246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:37.361549 22246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:37.361619 22246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:37.393599 22246 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:44417
I20260925 14:52:37.393668 22372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:44417 every 8 connection(s)
I20260925 14:52:37.394400 22246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:37.397049 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 22246
I20260925 14:52:37.397239 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:37.397867 17943 external_mini_cluster.cc:1428] Setting key 9c8fb977881ac784af2e60d750c0c16d
Sep 25 14:52:37 dist-test-slave-v5vl krb5kdc[22008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347957, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:37.415678 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.396638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:51115 (local address 127.17.133.254:45143)
0925 14:52:37.396795 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:37.396799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:37.398290 (+ 1491us) server_negotiation.cc:407] Connection header received
0925 14:52:37.400213 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:37.400218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:37.400296 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:37.400406 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:37.401865 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.402857 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.405295 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.406014 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.406156 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.409929 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:37.409940 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:37.409944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:37.409973 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:37.412750 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.413520 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.413525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.413528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.413712 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.414129 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.414133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.414136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.414387 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:37.414545 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:37.415186 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 14:52:37.415376 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
I20260925 14:52:37.416769 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.397190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45143 (local address 127.17.133.194:51115)
0925 14:52:37.398125 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.398171 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:37.399979 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:37.400672 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:37.400683 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:37.401160 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:37.401681 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.401702 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.403101 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.403106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.405110 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.405125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.406247 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.406255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.406549 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.407211 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:37.407239 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:37.409713 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:37.412913 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:37.412921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:37.412935 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:37.413401 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:37.413815 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:37.413822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:37.413828 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:37.414006 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:37.414498 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:37.414506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:37.414853 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 14:52:37.415103 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 14:52:37.415568 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":321,"threads_started":1}
I20260925 14:52:37.418020 22373 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
I20260925 14:52:37.418339 22373 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:37.418998 22373 heartbeater.cc:511] Master 127.17.133.254:45143 requested a full tablet report, sending...
I20260925 14:52:37.420428 22052 ts_manager.cc:194] Registered new tserver with Master: 4dbd17896d6144b0b7ae76bdc86948ae (127.17.133.194:44417)
I20260925 14:52:37.421690 22052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:51115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:52:37.439641 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45143
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:37.746685 22378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:37.746955 22378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:37.747128 22378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:37.754201 22378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:37.754340 22378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:37.754535 22378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:37.764180 22378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:37.765455 22378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:37.766753 22378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:37.778978 22386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:37.778944 22384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:37.779021 22383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:37.779755 22378 server_base.cc:1034] running on GCE node
I20260925 14:52:37.780300 22378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:37.781212 22378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:37.782400 22378 hybrid_clock.cc:648] HybridClock initialized: now 1790347957782349 us; error 46 us; skew 500 ppm
I20260925 14:52:37.782707 22378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:37 dist-test-slave-v5vl krb5kdc[22008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347957, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:37.791669 22378 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:52:37.792894 22378 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:37.793035 22378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:37.793337 22378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:37.796096 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d"
format_stamp: "Formatted at 2026-09-25 14:52:37 on dist-test-slave-v5vl"
server_key: "cd8490578a581bec143dcf8b651b31db"
server_key_iv: "1c301826cc1d11e83a776458015d449f"
server_key_version: "encryptionkey@0"
I20260925 14:52:37.796979 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d"
format_stamp: "Formatted at 2026-09-25 14:52:37 on dist-test-slave-v5vl"
server_key: "cd8490578a581bec143dcf8b651b31db"
server_key_iv: "1c301826cc1d11e83a776458015d449f"
server_key_version: "encryptionkey@0"
I20260925 14:52:37.801777 22378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:52:37.805068 22392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:37.806319 22378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 14:52:37.806490 22378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d"
format_stamp: "Formatted at 2026-09-25 14:52:37 on dist-test-slave-v5vl"
server_key: "cd8490578a581bec143dcf8b651b31db"
server_key_iv: "1c301826cc1d11e83a776458015d449f"
server_key_version: "encryptionkey@0"
I20260925 14:52:37.806699 22378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:37.835747 22378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:37.839905 22378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:37.840175 22378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:37.841068 22378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:37.842383 22378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:37.842496 22378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:37.842610 22378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:37.842674 22378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:37.874354 22378 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:35945
I20260925 14:52:37.874404 22504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:35945 every 8 connection(s)
I20260925 14:52:37.875212 22378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:37.877337 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 22378
I20260925 14:52:37.877532 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:37.878001 17943 external_mini_cluster.cc:1428] Setting key e7aeba7da07231c63e17e5a14f311bf1
Sep 25 14:52:37 dist-test-slave-v5vl krb5kdc[22008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347957, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:37.894470 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.877650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:48877 (local address 127.17.133.254:45143)
0925 14:52:37.877877 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:37.877882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:37.879229 (+ 1347us) server_negotiation.cc:407] Connection header received
0925 14:52:37.881131 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:37.881135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:37.881202 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:37.881277 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:37.882684 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.883393 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.884933 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.885341 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.885502 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.888652 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:37.888657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:37.888661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:37.888698 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:37.891431 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.892184 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.892192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.892198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.892320 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.892792 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.892798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.892802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.893086 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:37.893240 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:37.893913 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 14:52:37.894121 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 14:52:37.895466 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.878278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45143 (local address 127.17.133.195:48877)
0925 14:52:37.879077 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.879127 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:37.880891 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:37.881491 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:37.881498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:37.881982 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:37.882550 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.882567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.883527 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.883537 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.884797 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.884807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.885599 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.885605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.885821 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.886408 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:37.886432 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:37.888413 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:37.891585 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:37.891594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:37.891610 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:37.892067 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:37.892433 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:37.892439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:37.892442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:37.892685 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:37.893213 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:37.893221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:37.893558 (+ 337us) client_negotiation.cc:769] Sending connection context
0925 14:52:37.893829 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 14:52:37.894361 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":345,"threads_started":1}
I20260925 14:52:37.896833 22505 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45143
I20260925 14:52:37.897131 22505 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:37.897830 22505 heartbeater.cc:511] Master 127.17.133.254:45143 requested a full tablet report, sending...
I20260925 14:52:37.899237 22052 ts_manager.cc:194] Registered new tserver with Master: 58d0e9b76e994e3eb30fcfbd7ec6203d (127.17.133.195:35945)
I20260925 14:52:37.900023 22052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:48877
I20260925 14:52:37.905133 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:37.921640 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.909672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.17.133.254:45143)
0925 14:52:37.909873 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:37.909877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:37.910320 (+ 443us) server_negotiation.cc:407] Connection header received
0925 14:52:37.910439 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:37.910443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:37.910509 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:37.910587 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:37.911977 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.912770 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.914022 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.914484 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.914590 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.916768 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:37.916773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:37.916776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:37.916804 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:37.919420 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.919940 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.919944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.919946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.920073 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.920382 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.920384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.920386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.920741 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:37.920900 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:37.921144 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 14:52:37.921281 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 14:52:37.925347 22052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:37.931490 22052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:37.952901 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.942585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.17.133.194:44417)
0925 14:52:37.943205 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:37.943214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:37.944314 (+ 1100us) server_negotiation.cc:407] Connection header received
0925 14:52:37.944441 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:37.944452 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:37.944781 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:37.944993 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:37.945996 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.947487 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.950279 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.951719 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.951880 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.952063 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:37.952407 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 14:52:37.952566 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":320,"threads_started":1}
I20260925 14:52:37.952971 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.942366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:44417 (local address 127.0.0.1:55316)
0925 14:52:37.943853 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.943900 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:37.944102 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:37.945302 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:37.945313 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:37.945359 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:37.945822 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.945835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.947651 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.947656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.950118 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.950136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.951933 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.951940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.952156 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.952208 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 14:52:37.952342 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 14:52:37.952515 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":3,"thread_start_us":3138,"threads_started":1}
I20260925 14:52:37.954281 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.942963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:45335 (local address 127.0.0.1:51060)
0925 14:52:37.945733 (+ 2770us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.945763 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:37.945907 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:37.947013 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:37.947021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:37.947050 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:37.947555 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.947569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.949429 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.949439 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.951286 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.951298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.953380 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.953386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.953642 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.953664 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:52:37.953770 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:52:37.953897 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2678,"mutex_wait_us":52,"thread_start_us":394,"threads_started":1}
I20260925 14:52:37.954317 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.943600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.17.133.193:45335)
0925 14:52:37.945381 (+ 1781us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:37.945387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:37.945866 (+ 479us) server_negotiation.cc:407] Connection header received
0925 14:52:37.946620 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:37.946628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:37.946826 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:37.947000 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:37.947812 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.949276 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.951467 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.952848 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.953056 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.953233 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:37.953834 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 14:52:37.953991 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":1491,"threads_started":1}
I20260925 14:52:37.956501 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.946113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:35945 (local address 127.0.0.1:34878)
0925 14:52:37.946689 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.946711 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:37.946837 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:37.949188 (+ 2351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:37.949194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:37.949220 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:37.949640 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.949649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.951495 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.951500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.953631 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.953645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.955747 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.955753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.955908 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.955943 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:52:37.956041 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 14:52:37.956156 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":4,"thread_start_us":268,"threads_started":1}
I20260925 14:52:37.956575 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.946217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.17.133.195:35945)
0925 14:52:37.948638 (+ 2421us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:37.948646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:37.948675 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:52:37.948781 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:37.948787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:37.948983 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:37.949179 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:37.949829 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.951319 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.953786 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.955388 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.955581 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.955812 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:37.956102 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 14:52:37.956268 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2204,"thread_start_us":2035,"threads_started":1}
I20260925 14:52:37.957252 22308 tablet_service.cc:1467] Processing CreateTablet for tablet 1adebed71e2e46f8b892031543de55f2 (DEFAULT_TABLE table=test-table [id=66c1c62599f2465fa752c3c29543725a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:37.957943 22240 heartbeater.cc:503] Master 127.17.133.254:45143 was elected leader, sending a full tablet report...
I20260925 14:52:37.957737 22175 tablet_service.cc:1467] Processing CreateTablet for tablet 1adebed71e2e46f8b892031543de55f2 (DEFAULT_TABLE table=test-table [id=66c1c62599f2465fa752c3c29543725a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:37.958881 22308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1adebed71e2e46f8b892031543de55f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:37.959331 22175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1adebed71e2e46f8b892031543de55f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:37.960246 22440 tablet_service.cc:1467] Processing CreateTablet for tablet 1adebed71e2e46f8b892031543de55f2 (DEFAULT_TABLE table=test-table [id=66c1c62599f2465fa752c3c29543725a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:37.961455 22440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1adebed71e2e46f8b892031543de55f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:37.970018 22523 tablet_bootstrap.cc:492] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: Bootstrap starting.
I20260925 14:52:37.972947 22524 tablet_bootstrap.cc:492] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: Bootstrap starting.
I20260925 14:52:37.974608 22523 tablet_bootstrap.cc:654] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:37.975162 22525 tablet_bootstrap.cc:492] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: Bootstrap starting.
I20260925 14:52:37.975900 22523 log.cc:824] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:37.976688 22524 tablet_bootstrap.cc:654] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:37.977818 22524 log.cc:824] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:37.978228 22525 tablet_bootstrap.cc:654] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:37.979110 22525 log.cc:824] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:37.979125 22523 tablet_bootstrap.cc:492] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: No bootstrap required, opened a new log
I20260925 14:52:37.979439 22523 ts_tablet_manager.cc:1397] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 14:52:37.981851 22524 tablet_bootstrap.cc:492] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: No bootstrap required, opened a new log
I20260925 14:52:37.982123 22524 ts_tablet_manager.cc:1397] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 14:52:37.982404 22525 tablet_bootstrap.cc:492] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: No bootstrap required, opened a new log
I20260925 14:52:37.982648 22525 ts_tablet_manager.cc:1397] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 14:52:37.990061 22525 raft_consensus.cc:348] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } }
I20260925 14:52:37.990447 22525 raft_consensus.cc:374] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:37.990556 22525 raft_consensus.cc:729] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d0e9b76e994e3eb30fcfbd7ec6203d, State: Initialized, Role: FOLLOWER
I20260925 14:52:37.990398 22523 raft_consensus.cc:348] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } }
I20260925 14:52:37.990854 22523 raft_consensus.cc:374] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:37.990957 22523 raft_consensus.cc:729] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbd17896d6144b0b7ae76bdc86948ae, State: Initialized, Role: FOLLOWER
I20260925 14:52:37.991361 22525 consensus_queue.cc:260] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } }
I20260925 14:52:37.991547 22523 consensus_queue.cc:260] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } }
I20260925 14:52:37.992750 22524 raft_consensus.cc:348] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } }
I20260925 14:52:37.993398 22524 raft_consensus.cc:374] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:37.993553 22524 raft_consensus.cc:729] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfacbd1b680b4b3d9c080e406a7c6ced, State: Initialized, Role: FOLLOWER
I20260925 14:52:37.994396 22505 heartbeater.cc:503] Master 127.17.133.254:45143 was elected leader, sending a full tablet report...
I20260925 14:52:37.994872 22525 ts_tablet_manager.cc:1428] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260925 14:52:37.995651 22373 heartbeater.cc:503] Master 127.17.133.254:45143 was elected leader, sending a full tablet report...
I20260925 14:52:37.996343 22523 ts_tablet_manager.cc:1428] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
I20260925 14:52:37.996245 22524 consensus_queue.cc:260] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } }
I20260925 14:52:37.998164 22524 ts_tablet_manager.cc:1428] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260925 14:52:38.009557 22531 raft_consensus.cc:482] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:38.009837 22531 raft_consensus.cc:504] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } }
I20260925 14:52:38.011662 22531 leader_election.cc:288] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dbd17896d6144b0b7ae76bdc86948ae (127.17.133.194:44417), 58d0e9b76e994e3eb30fcfbd7ec6203d (127.17.133.195:35945)
I20260925 14:52:38.018632 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.012022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:35779 (local address 127.17.133.194:44417)
0925 14:52:38.012248 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:38.012255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:38.012814 (+ 559us) server_negotiation.cc:407] Connection header received
0925 14:52:38.013184 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:38.013191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:38.013273 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:38.013340 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:38.014114 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.014910 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.016695 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.017704 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.017905 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:38.017980 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:38.018263 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 14:52:38.018383 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:52:38.018755 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.012023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:44417 (local address 127.17.133.193:35779)
0925 14:52:38.012682 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:38.012729 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:38.012909 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:38.013544 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:38.013548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:38.013568 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:38.013972 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:38.013984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.015037 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.015044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:38.016516 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:38.016527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.017918 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.017924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:38.018109 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:38.018131 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:52:38.018246 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 14:52:38.018370 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":328,"threads_started":1}
I20260925 14:52:38.018944 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.012905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:54857 (local address 127.17.133.195:35945)
0925 14:52:38.013157 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:38.013162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:38.013515 (+ 353us) server_negotiation.cc:407] Connection header received
0925 14:52:38.013671 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:38.013675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:38.013729 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:38.013811 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:38.014411 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.015185 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.016998 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.018016 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.018215 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:38.018292 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:38.018578 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 14:52:38.018699 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:52:38.019107 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.012805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:35945 (local address 127.17.133.193:54857)
0925 14:52:38.013400 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:38.013416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:38.013547 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:38.013877 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:38.013883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:38.013913 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:38.014259 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:38.014267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.015323 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.015329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:38.016846 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:38.016858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.018265 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.018271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:38.018409 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:38.018430 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 14:52:38.018563 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 14:52:38.018684 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":1,"thread_start_us":299,"threads_started":1}
I20260925 14:52:38.019913 22328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1adebed71e2e46f8b892031543de55f2" candidate_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" is_pre_election: true
I20260925 14:52:38.020015 22460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1adebed71e2e46f8b892031543de55f2" candidate_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" is_pre_election: true
I20260925 14:52:38.020417 22328 raft_consensus.cc:2454] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfacbd1b680b4b3d9c080e406a7c6ced in term 0.
I20260925 14:52:38.020495 22460 raft_consensus.cc:2454] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfacbd1b680b4b3d9c080e406a7c6ced in term 0.
I20260925 14:52:38.021106 22131 leader_election.cc:302] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbd17896d6144b0b7ae76bdc86948ae, dfacbd1b680b4b3d9c080e406a7c6ced; no voters:
I20260925 14:52:38.021471 22531 raft_consensus.cc:2781] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:38.021631 22531 raft_consensus.cc:482] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:38.021708 22531 raft_consensus.cc:3037] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:38.023995 22531 raft_consensus.cc:504] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } }
I20260925 14:52:38.024564 22531 leader_election.cc:288] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [CANDIDATE]: Term 1 election: Requested vote from peers 4dbd17896d6144b0b7ae76bdc86948ae (127.17.133.194:44417), 58d0e9b76e994e3eb30fcfbd7ec6203d (127.17.133.195:35945)
I20260925 14:52:38.025110 22328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1adebed71e2e46f8b892031543de55f2" candidate_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbd17896d6144b0b7ae76bdc86948ae"
I20260925 14:52:38.025193 22460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1adebed71e2e46f8b892031543de55f2" candidate_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d"
I20260925 14:52:38.025285 22328 raft_consensus.cc:3037] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:38.025408 22460 raft_consensus.cc:3037] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:38.027491 22328 raft_consensus.cc:2454] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfacbd1b680b4b3d9c080e406a7c6ced in term 1.
I20260925 14:52:38.027495 22460 raft_consensus.cc:2454] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfacbd1b680b4b3d9c080e406a7c6ced in term 1.
I20260925 14:52:38.027814 22131 leader_election.cc:302] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbd17896d6144b0b7ae76bdc86948ae, dfacbd1b680b4b3d9c080e406a7c6ced; no voters:
I20260925 14:52:38.028041 22531 raft_consensus.cc:2781] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:38.028481 22531 raft_consensus.cc:686] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 1 LEADER]: Becoming Leader. State: Replica: dfacbd1b680b4b3d9c080e406a7c6ced, State: Running, Role: LEADER
I20260925 14:52:38.028954 22531 consensus_queue.cc:237] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } }
I20260925 14:52:38.033383 22052 catalog_manager.cc:5495] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced reported cstate change: term changed from 0 to 1, leader changed from <none> to dfacbd1b680b4b3d9c080e406a7c6ced (127.17.133.193). New cstate: current_term: 1 leader_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfacbd1b680b4b3d9c080e406a7c6ced" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 45335 } health_report { overall_health: HEALTHY } } }
I20260925 14:52:38.052328 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.047734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.17.133.193:45335)
0925 14:52:38.047893 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:38.047897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:38.047912 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:52:38.047982 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:38.047985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:38.048049 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:38.048164 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:38.048452 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.049193 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.050439 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.050811 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.050892 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:38.051280 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:38.051733 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:38.051833 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:38.051977 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 14:52:38.052034 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:52:38.060315 22460 raft_consensus.cc:1261] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [term 1 FOLLOWER]: Refusing update from remote peer dfacbd1b680b4b3d9c080e406a7c6ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:38.060313 22328 raft_consensus.cc:1261] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [term 1 FOLLOWER]: Refusing update from remote peer dfacbd1b680b4b3d9c080e406a7c6ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:38.061262 22534 consensus_queue.cc:1035] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d0e9b76e994e3eb30fcfbd7ec6203d" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 35945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:38.061642 22531 consensus_queue.cc:1035] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dbd17896d6144b0b7ae76bdc86948ae" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:38.089102 22052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:38.089315 22052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:38.092857 22052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76335e76149842cba144ace969eeb9e6: Sending DeleteTablet for 3 replicas of tablet 1adebed71e2e46f8b892031543de55f2
I20260925 14:52:38.094003 22308 tablet_service.cc:1514] Processing DeleteTablet for tablet 1adebed71e2e46f8b892031543de55f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:38 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:55316
I20260925 14:52:38.094080 22440 tablet_service.cc:1514] Processing DeleteTablet for tablet 1adebed71e2e46f8b892031543de55f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:38 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:34878
I20260925 14:52:38.094437 22175 tablet_service.cc:1514] Processing DeleteTablet for tablet 1adebed71e2e46f8b892031543de55f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:38 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:51060
I20260925 14:52:38.097394 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 22113
I20260925 14:52:38.109436 22548 tablet_replica.cc:330] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: stopping tablet replica
I20260925 14:52:38.110389 22549 tablet_replica.cc:330] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: stopping tablet replica
I20260925 14:52:38.110842 22549 raft_consensus.cc:2229] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:38.111138 22548 raft_consensus.cc:2229] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:38.111713 22549 raft_consensus.cc:2258] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:38.112344 22548 raft_consensus.cc:2258] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:38.112490 22547 tablet_replica.cc:330] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: stopping tablet replica
I20260925 14:52:38.112982 22547 raft_consensus.cc:2229] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:38.113489 22547 raft_consensus.cc:2258] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:38.115324 22548 ts_tablet_manager.cc:1905] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:38.116461 22549 ts_tablet_manager.cc:1905] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:38.118505 22547 ts_tablet_manager.cc:1905] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 14:52:38 dist-test-slave-v5vl krb5kdc[22008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:38.122274 22548 ts_tablet_manager.cc:1918] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:38.122481 22548 log.cc:1196] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/1adebed71e2e46f8b892031543de55f2
I20260925 14:52:38.123073 22548 ts_tablet_manager.cc:1939] T 1adebed71e2e46f8b892031543de55f2 P 58d0e9b76e994e3eb30fcfbd7ec6203d: Deleting consensus metadata
I20260925 14:52:38.123822 22549 ts_tablet_manager.cc:1918] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:38.124013 22549 log.cc:1196] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/1adebed71e2e46f8b892031543de55f2
I20260925 14:52:38.124668 22549 ts_tablet_manager.cc:1939] T 1adebed71e2e46f8b892031543de55f2 P dfacbd1b680b4b3d9c080e406a7c6ced: Deleting consensus metadata
I20260925 14:52:38.125007 22039 catalog_manager.cc:4841] TS 58d0e9b76e994e3eb30fcfbd7ec6203d (127.17.133.195:35945): tablet 1adebed71e2e46f8b892031543de55f2 (table test-table [id=66c1c62599f2465fa752c3c29543725a]) successfully deleted
I20260925 14:52:38.126374 22037 catalog_manager.cc:4841] TS dfacbd1b680b4b3d9c080e406a7c6ced (127.17.133.193:45335): tablet 1adebed71e2e46f8b892031543de55f2 (table test-table [id=66c1c62599f2465fa752c3c29543725a]) successfully deleted
I20260925 14:52:38.127327 22547 ts_tablet_manager.cc:1918] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:38.127487 22547 log.cc:1196] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/1adebed71e2e46f8b892031543de55f2
I20260925 14:52:38.127900 22547 ts_tablet_manager.cc:1939] T 1adebed71e2e46f8b892031543de55f2 P 4dbd17896d6144b0b7ae76bdc86948ae: Deleting consensus metadata
I20260925 14:52:38.129254 22039 catalog_manager.cc:4841] TS 4dbd17896d6144b0b7ae76bdc86948ae (127.17.133.194:44417): tablet 1adebed71e2e46f8b892031543de55f2 (table test-table [id=66c1c62599f2465fa752c3c29543725a]) successfully deleted
I20260925 14:52:38.130081 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.099264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.17.133.193:45335)
0925 14:52:38.099490 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:38.099500 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:38.109631 (+ 10131us) server_negotiation.cc:407] Connection header received
0925 14:52:38.109863 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:38.109873 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:38.109993 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:38.110109 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:38.111854 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.113112 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.115244 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.115910 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.116033 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:38.121311 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:38.121324 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:38.121340 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:38.121394 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:38.125572 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:38.126910 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:38.126917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:38.126921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:38.127099 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:38.127535 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:38.127540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:38.127543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:38.127947 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:38.128241 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:38.129461 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0925 14:52:38.129694 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:52:38.130803 22235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:38.229893 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 22113
I20260925 14:52:38.247871 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 22246
Sep 25 14:52:38 dist-test-slave-v5vl krb5kdc[22008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:52:38.260952 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.248647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.17.133.194:44417)
0925 14:52:38.248817 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:38.248821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:38.248838 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:52:38.248973 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:38.248977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:38.249032 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:38.249119 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:38.250237 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.250946 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.251991 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.252319 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.252399 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:38.256000 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:38.256010 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:38.256024 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:38.256050 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:38.258497 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:38.259042 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:38.259048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:38.259050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:38.259230 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:38.259621 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:38.259624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:38.259626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:38.259944 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:38.260143 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:38.260384 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 14:52:38.260526 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:52:38.261574 22368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:38.348089 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 22246
I20260925 14:52:38.366657 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 22378
Sep 25 14:52:38 dist-test-slave-v5vl krb5kdc[22008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:52:38.381373 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.367491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.17.133.195:35945)
0925 14:52:38.367690 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:38.367696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:38.367718 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:52:38.367850 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:38.367855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:38.367931 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:38.368030 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:38.369266 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.370334 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.371463 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:38.371820 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:38.371908 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:38.376022 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:38.376035 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:38.376051 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:38.376088 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:38.378688 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:38.379349 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:38.379357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:38.379359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:38.379487 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:38.379840 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:38.379846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:38.379849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:38.380187 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:38.380474 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:38.380875 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 14:52:38.381083 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:52:38.381958 22500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:38.470436 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 22378
I20260925 14:52:38.486496 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 22021
I20260925 14:52:38.487558 22082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:38.593176 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 22021
2026-09-25T14:52:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3915 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T14:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:38Z Disabled control of system clock
I20260925 14:52:38.657346 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:44395
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:44395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:52:38.964751 22558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:38.965021 22558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:38.965101 22558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:38.972105 22558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:38.972203 22558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:38.972247 22558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:38.972304 22558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:38.972347 22558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:38.981235 22558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:44395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:44395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:38.982477 22558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:38.984117 22558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:38.995203 22564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:38.995227 22563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:38.995174 22566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:38.996781 22558 server_base.cc:1034] running on GCE node
I20260925 14:52:38.997466 22558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:38.998675 22558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:38.999867 22558 hybrid_clock.cc:648] HybridClock initialized: now 1790347958999835 us; error 25 us; skew 500 ppm
I20260925 14:52:39.000191 22558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:52:39.002323 22558 webserver.cc:460] Webserver started at http://127.17.133.254:37877/ using document root <none> and password file <none>
I20260925 14:52:39.002985 22558 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:39.003094 22558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:39.003419 22558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:39.006294 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "83044b0cda16414d83e58177f1a0ee0a"
format_stamp: "Formatted at 2026-09-25 14:52:38 on dist-test-slave-v5vl"
server_key: "a6097db432eb27752669aac07f811e51"
server_key_iv: "3c50c995f4bd660fd6be04ce705d41d7"
server_key_version: "encryptionkey@0"
I20260925 14:52:39.007053 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "83044b0cda16414d83e58177f1a0ee0a"
format_stamp: "Formatted at 2026-09-25 14:52:38 on dist-test-slave-v5vl"
server_key: "a6097db432eb27752669aac07f811e51"
server_key_iv: "3c50c995f4bd660fd6be04ce705d41d7"
server_key_version: "encryptionkey@0"
I20260925 14:52:39.011832 22558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 14:52:39.015072 22572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:39.016297 22558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:39.016456 22558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "83044b0cda16414d83e58177f1a0ee0a"
format_stamp: "Formatted at 2026-09-25 14:52:38 on dist-test-slave-v5vl"
server_key: "a6097db432eb27752669aac07f811e51"
server_key_iv: "3c50c995f4bd660fd6be04ce705d41d7"
server_key_version: "encryptionkey@0"
I20260925 14:52:39.016692 22558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:39.022038 22558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:39.022764 22558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:39.022998 22558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:39.040319 22558 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:44395
I20260925 14:52:39.040325 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:44395 every 8 connection(s)
I20260925 14:52:39.041610 22558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:39.042922 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 22558
I20260925 14:52:39.043143 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:39.043916 17943 external_mini_cluster.cc:1428] Setting key 8c23579e18c10d5f0c4380ea55ab347b
I20260925 14:52:39.046051 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:39.051899 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:39.045940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.17.133.254:44395)
0925 14:52:39.046831 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:39.046840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:39.046878 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:52:39.048409 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:39.048428 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:39.048887 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:39.049279 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:39.050283 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:39.050292 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:39.050304 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:39.050319 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:39.050391 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:39.050407 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:39.050638 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:39.050797 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:52:39.051032 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":326,"threads_started":1}
I20260925 14:52:39.057457 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a: Bootstrap starting.
I20260925 14:52:39.060191 22624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:39.061373 22624 log.cc:824] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:39.063884 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a: No bootstrap required, opened a new log
I20260925 14:52:39.070542 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83044b0cda16414d83e58177f1a0ee0a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44395 } }
I20260925 14:52:39.070865 22624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:39.071000 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83044b0cda16414d83e58177f1a0ee0a, State: Initialized, Role: FOLLOWER
I20260925 14:52:39.071542 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83044b0cda16414d83e58177f1a0ee0a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44395 } }
I20260925 14:52:39.071748 22624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:39.071857 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:39.072016 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:39.074081 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83044b0cda16414d83e58177f1a0ee0a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44395 } }
I20260925 14:52:39.074544 22624 leader_election.cc:302] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83044b0cda16414d83e58177f1a0ee0a; no voters:
I20260925 14:52:39.075070 22624 leader_election.cc:288] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:39.075160 22629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:39.075822 22629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [term 1 LEADER]: Becoming Leader. State: Replica: 83044b0cda16414d83e58177f1a0ee0a, State: Running, Role: LEADER
I20260925 14:52:39.076267 22629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83044b0cda16414d83e58177f1a0ee0a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44395 } }
I20260925 14:52:39.076999 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:39.078924 22631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83044b0cda16414d83e58177f1a0ee0a. Latest consensus state: current_term: 1 leader_uuid: "83044b0cda16414d83e58177f1a0ee0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83044b0cda16414d83e58177f1a0ee0a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44395 } } }
I20260925 14:52:39.079206 22631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:39.079890 22630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83044b0cda16414d83e58177f1a0ee0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83044b0cda16414d83e58177f1a0ee0a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44395 } } }
I20260925 14:52:39.080171 22630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:39.081562 22636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:39.087119 22636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:39.095093 22636 catalog_manager.cc:1347] Generated new cluster ID: 0994cdce6ca3455d9b3220a9d16e2400
I20260925 14:52:39.095204 22636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:39.114416 22636 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:39.114833 22636 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:39.122987 22636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a: Generated new TSK 0
I20260925 14:52:39.123673 22636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:52:39.132256 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:44395
--builtin_ntp_servers=127.17.133.212:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:52:39.460390 22648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:39.460749 22648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:39.460925 22648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:39.468237 22648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:39.468386 22648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:39.468573 22648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:39.478034 22648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:44395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:39.479382 22648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:39.481220 22648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:39.494278 22654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:39.494278 22653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:39.494251 22656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:39.494989 22648 server_base.cc:1034] running on GCE node
I20260925 14:52:39.495551 22648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:39.496374 22648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:39.497560 22648 hybrid_clock.cc:648] HybridClock initialized: now 1790347959497512 us; error 37 us; skew 500 ppm
I20260925 14:52:39.497821 22648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:52:39.499840 22648 webserver.cc:460] Webserver started at http://127.17.133.193:43227/ using document root <none> and password file <none>
I20260925 14:52:39.500481 22648 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:39.500563 22648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:39.500898 22648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:39.503616 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "dbef1eae066f4bd994c55711819e6511"
format_stamp: "Formatted at 2026-09-25 14:52:39 on dist-test-slave-v5vl"
server_key: "839903158e2e7970322e7f322396cf98"
server_key_iv: "5b0fc5f57415074a9a48d4c55a37ae2e"
server_key_version: "encryptionkey@0"
I20260925 14:52:39.504305 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "dbef1eae066f4bd994c55711819e6511"
format_stamp: "Formatted at 2026-09-25 14:52:39 on dist-test-slave-v5vl"
server_key: "839903158e2e7970322e7f322396cf98"
server_key_iv: "5b0fc5f57415074a9a48d4c55a37ae2e"
server_key_version: "encryptionkey@0"
I20260925 14:52:39.509042 22648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260925 14:52:39.512141 22662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:39.513447 22648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:39.513612 22648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "dbef1eae066f4bd994c55711819e6511"
format_stamp: "Formatted at 2026-09-25 14:52:39 on dist-test-slave-v5vl"
server_key: "839903158e2e7970322e7f322396cf98"
server_key_iv: "5b0fc5f57415074a9a48d4c55a37ae2e"
server_key_version: "encryptionkey@0"
I20260925 14:52:39.513841 22648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:39.518996 22648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:39.519919 22648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:39.520172 22648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:39.521188 22648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:39.522558 22648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:39.522642 22648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:39.522792 22648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:39.522890 22648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:39.554423 22648 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:43955
I20260925 14:52:39.554476 22774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:43955 every 8 connection(s)
I20260925 14:52:39.555694 22648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:39.563556 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:39.558184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:35933 (local address 127.17.133.254:44395)
0925 14:52:39.558584 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:39.558589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:39.559733 (+ 1144us) server_negotiation.cc:407] Connection header received
0925 14:52:39.561338 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:39.561342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:39.561396 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:39.561487 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:39.562823 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:39.562827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:39.562830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:39.562844 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:39.562879 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:39.562890 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:39.562973 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:39.563180 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 14:52:39.563243 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":215,"threads_started":1}
I20260925 14:52:39.563561 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 22648
I20260925 14:52:39.563768 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:39.564301 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:39.558758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:44395 (local address 127.17.133.193:35933)
0925 14:52:39.559585 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:39.559622 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:39.561116 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:39.561699 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:39.561711 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:39.561823 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:39.562482 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:39.562484 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:39.562538 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:39.562652 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:39.562966 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:39.562973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:39.562978 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:52:39.563175 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 14:52:39.563349 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":244,"threads_started":1}
I20260925 14:52:39.564390 17943 external_mini_cluster.cc:1428] Setting key a9b3293fa404535a1804551809bce5b2
I20260925 14:52:39.565956 22775 heartbeater.cc:347] Connected to a master server at 127.17.133.254:44395
I20260925 14:52:39.566234 22775 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:39.566833 22775 heartbeater.cc:511] Master 127.17.133.254:44395 requested a full tablet report, sending...
I20260925 14:52:39.567605 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:44395
--builtin_ntp_servers=127.17.133.212:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 14:52:39.569906 22588 ts_manager.cc:194] Registered new tserver with Master: dbef1eae066f4bd994c55711819e6511 (127.17.133.193:43955)
W20260925 14:52:39.876771 22779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:39.877096 22779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:39.877310 22779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:39.884167 22779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:39.884274 22779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:39.884450 22779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:39.893018 22779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:44395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:39.894255 22779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:39.895903 22779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:39.908015 22784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:39.908052 22787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:39.908093 22785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:39.909049 22779 server_base.cc:1034] running on GCE node
I20260925 14:52:39.909583 22779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:39.910445 22779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:39.911665 22779 hybrid_clock.cc:648] HybridClock initialized: now 1790347959911622 us; error 56 us; skew 500 ppm
I20260925 14:52:39.911948 22779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:52:39.914085 22779 webserver.cc:460] Webserver started at http://127.17.133.194:45097/ using document root <none> and password file <none>
I20260925 14:52:39.914738 22779 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:39.914862 22779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:39.915180 22779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:39.917927 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "f2f0f44dcfa4463bb2c5fd4adae14847"
format_stamp: "Formatted at 2026-09-25 14:52:39 on dist-test-slave-v5vl"
server_key: "99825aa24efa061a6ae4d95553862e3b"
server_key_iv: "8d851025ca0db57ef5614148ea22e4ff"
server_key_version: "encryptionkey@0"
I20260925 14:52:39.918720 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "f2f0f44dcfa4463bb2c5fd4adae14847"
format_stamp: "Formatted at 2026-09-25 14:52:39 on dist-test-slave-v5vl"
server_key: "99825aa24efa061a6ae4d95553862e3b"
server_key_iv: "8d851025ca0db57ef5614148ea22e4ff"
server_key_version: "encryptionkey@0"
I20260925 14:52:39.923287 22779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 14:52:39.926131 22793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:39.927263 22779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:39.927435 22779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "f2f0f44dcfa4463bb2c5fd4adae14847"
format_stamp: "Formatted at 2026-09-25 14:52:39 on dist-test-slave-v5vl"
server_key: "99825aa24efa061a6ae4d95553862e3b"
server_key_iv: "8d851025ca0db57ef5614148ea22e4ff"
server_key_version: "encryptionkey@0"
I20260925 14:52:39.927654 22779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:39.933030 22779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:39.933761 22779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:39.934044 22779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:39.934888 22779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:39.936172 22779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:39.936286 22779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:39.936414 22779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:39.936497 22779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:39.967315 22779 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:40967
I20260925 14:52:39.967368 22905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:40967 every 8 connection(s)
I20260925 14:52:39.968604 22779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:39.975291 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 22779
I20260925 14:52:39.975463 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:39.976076 17943 external_mini_cluster.cc:1428] Setting key b3a8708864d02c3040cef37f79ac0411
I20260925 14:52:39.976411 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:39.971002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:34645 (local address 127.17.133.254:44395)
0925 14:52:39.971181 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:39.971186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:39.972531 (+ 1345us) server_negotiation.cc:407] Connection header received
0925 14:52:39.974163 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:39.974167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:39.974221 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:39.974306 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:39.975684 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:39.975690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:39.975694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:39.975715 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:39.975752 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:39.975768 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:39.975884 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:39.976055 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 14:52:39.976118 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:52:39.977269 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:39.971541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:44395 (local address 127.17.133.194:34645)
0925 14:52:39.972394 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:39.972429 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:39.973910 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:39.974502 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:39.974509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:39.974594 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:39.975285 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:39.975288 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:39.975323 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:39.975454 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:39.975863 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:39.975870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:39.975875 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:52:39.976104 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 14:52:39.976353 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":259,"threads_started":1}
I20260925 14:52:39.978562 22906 heartbeater.cc:347] Connected to a master server at 127.17.133.254:44395
I20260925 14:52:39.978856 22906 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:39.978932 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:44395
--builtin_ntp_servers=127.17.133.212:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 14:52:39.979429 22906 heartbeater.cc:511] Master 127.17.133.254:44395 requested a full tablet report, sending...
I20260925 14:52:39.980758 22588 ts_manager.cc:194] Registered new tserver with Master: f2f0f44dcfa4463bb2c5fd4adae14847 (127.17.133.194:40967)
W20260925 14:52:40.306769 22909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:40.307030 22909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:40.307114 22909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:40.313853 22909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:40.313932 22909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:40.314121 22909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:40.323058 22909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:44395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:40.324241 22909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:40.326027 22909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:40.338299 22914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:40.338240 22915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:40.338331 22917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:40.339097 22909 server_base.cc:1034] running on GCE node
I20260925 14:52:40.339565 22909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:40.340379 22909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:40.341593 22909 hybrid_clock.cc:648] HybridClock initialized: now 1790347960341538 us; error 50 us; skew 500 ppm
I20260925 14:52:40.341899 22909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:52:40.343963 22909 webserver.cc:460] Webserver started at http://127.17.133.195:42221/ using document root <none> and password file <none>
I20260925 14:52:40.344668 22909 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:40.344775 22909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:40.345129 22909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:40.347831 22909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "9feb7aca38094f7f85a058226e3e696c"
format_stamp: "Formatted at 2026-09-25 14:52:40 on dist-test-slave-v5vl"
server_key: "a7dca041196d8d3cc886a84d53db83c2"
server_key_iv: "75991586e96ffe1b2ce2518f1fe5e5a3"
server_key_version: "encryptionkey@0"
I20260925 14:52:40.348578 22909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "9feb7aca38094f7f85a058226e3e696c"
format_stamp: "Formatted at 2026-09-25 14:52:40 on dist-test-slave-v5vl"
server_key: "a7dca041196d8d3cc886a84d53db83c2"
server_key_iv: "75991586e96ffe1b2ce2518f1fe5e5a3"
server_key_version: "encryptionkey@0"
I20260925 14:52:40.353497 22909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:52:40.356775 22923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:40.358008 22909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:52:40.358181 22909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "9feb7aca38094f7f85a058226e3e696c"
format_stamp: "Formatted at 2026-09-25 14:52:40 on dist-test-slave-v5vl"
server_key: "a7dca041196d8d3cc886a84d53db83c2"
server_key_iv: "75991586e96ffe1b2ce2518f1fe5e5a3"
server_key_version: "encryptionkey@0"
I20260925 14:52:40.358392 22909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:40.364159 22909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:40.364923 22909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:40.365173 22909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:40.365988 22909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:40.367214 22909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:40.367331 22909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:40.367458 22909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:40.367544 22909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:40.398216 22909 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:46677
I20260925 14:52:40.398263 23035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:46677 every 8 connection(s)
I20260925 14:52:40.399398 22909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:40.406658 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.401806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:59143 (local address 127.17.133.254:44395)
0925 14:52:40.401968 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.401972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.403229 (+ 1257us) server_negotiation.cc:407] Connection header received
0925 14:52:40.404790 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.404794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.404842 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.404927 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.406089 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.406092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.406095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.406107 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.406129 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.406139 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.406224 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.406379 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.406446 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:52:40.407421 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 22909
I20260925 14:52:40.407603 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:40.407714 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.402283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:44395 (local address 127.17.133.195:59143)
0925 14:52:40.403076 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.403112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:40.404507 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:40.405103 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:40.405110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:40.405190 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:40.405849 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:40.405852 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:40.405886 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:40.405960 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:40.406214 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:40.406220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:40.406225 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:52:40.406439 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 14:52:40.406612 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":230,"threads_started":1}
I20260925 14:52:40.408150 17943 external_mini_cluster.cc:1428] Setting key 8df68a6b3347a716e2ac826779f1a9e8
I20260925 14:52:40.409060 23036 heartbeater.cc:347] Connected to a master server at 127.17.133.254:44395
I20260925 14:52:40.409370 23036 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:40.409973 23036 heartbeater.cc:511] Master 127.17.133.254:44395 requested a full tablet report, sending...
I20260925 14:52:40.411413 22588 ts_manager.cc:194] Registered new tserver with Master: 9feb7aca38094f7f85a058226e3e696c (127.17.133.195:46677)
I20260925 14:52:40.423317 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:40.429735 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.427496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.17.133.254:44395)
0925 14:52:40.427671 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.427675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.428208 (+ 533us) server_negotiation.cc:407] Connection header received
0925 14:52:40.428331 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.428335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.428379 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.428458 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.429291 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.429294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.429297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.429310 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.429337 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.429347 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.429435 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.429511 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.429549 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:52:40.433459 22588 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:40.440011 22588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:40.466799 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.451021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.17.133.193:43955)
0925 14:52:40.462085 (+ 11064us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.462098 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.463243 (+ 1145us) server_negotiation.cc:407] Connection header received
0925 14:52:40.463485 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.463493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.463700 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.463814 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.465849 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.465858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.465872 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.465897 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.465952 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.465972 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.466077 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.466371 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.466477 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10835,"thread_start_us":11066,"threads_started":1}
I20260925 14:52:40.466864 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.450589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:40967 (local address 127.0.0.1:52714)
0925 14:52:40.461645 (+ 11056us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.461693 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:40.461860 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:40.464532 (+ 2672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:40.464539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:40.464649 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:40.465554 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:40.465560 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:40.465593 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:40.465676 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:40.466167 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:40.466178 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:40.466185 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:52:40.466347 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 14:52:40.466437 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10708,"thread_start_us":12298,"threads_started":1}
I20260925 14:52:40.467620 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.450775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.17.133.194:40967)
0925 14:52:40.462828 (+ 12053us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.462838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.462872 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:52:40.462956 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.462965 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.463225 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.463373 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.465854 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.465860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.465871 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.465897 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.465962 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.465980 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.466121 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.467184 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.467270 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11890,"thread_start_us":11778,"threads_started":1}
I20260925 14:52:40.467664 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.450719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43955 (local address 127.0.0.1:38422)
0925 14:52:40.463131 (+ 12412us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.463148 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:40.463327 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:40.464698 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:40.464707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:40.464733 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:40.465649 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:40.465652 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:40.465663 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:40.465689 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:40.466167 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:40.466178 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:40.466185 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:52:40.467186 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0925 14:52:40.467234 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12308,"thread_start_us":11282,"threads_started":1}
I20260925 14:52:40.471190 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.463148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:46677 (local address 127.0.0.1:55682)
0925 14:52:40.467835 (+ 4687us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.467852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:40.467929 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:40.469216 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:40.469220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:40.469259 (+ 39us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:40.469966 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:40.469972 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:40.469991 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:40.470040 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:40.470782 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:40.470786 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:40.470791 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:52:40.470909 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 14:52:40.470954 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3833,"thread_start_us":5732,"threads_started":1}
I20260925 14:52:40.471423 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.463199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.17.133.195:46677)
0925 14:52:40.468784 (+ 5585us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.468791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.468821 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:52:40.468888 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.468894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.469078 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.469224 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.470542 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.470550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.470566 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.470584 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.470638 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.470658 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.470830 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.470957 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.471043 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5430,"thread_start_us":5251,"threads_started":1}
I20260925 14:52:40.472177 22841 tablet_service.cc:1467] Processing CreateTablet for tablet 41159abc074a45708473401ad1f288b7 (DEFAULT_TABLE table=test-table [id=69ba551a012a4efd9e48299c5e2c23a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:40.473078 22710 tablet_service.cc:1467] Processing CreateTablet for tablet 41159abc074a45708473401ad1f288b7 (DEFAULT_TABLE table=test-table [id=69ba551a012a4efd9e48299c5e2c23a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:40.473788 22841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41159abc074a45708473401ad1f288b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:40.474164 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41159abc074a45708473401ad1f288b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:40.474901 22971 tablet_service.cc:1467] Processing CreateTablet for tablet 41159abc074a45708473401ad1f288b7 (DEFAULT_TABLE table=test-table [id=69ba551a012a4efd9e48299c5e2c23a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:40.476127 22971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41159abc074a45708473401ad1f288b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:40.486300 23054 tablet_bootstrap.cc:492] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511: Bootstrap starting.
I20260925 14:52:40.488940 23055 tablet_bootstrap.cc:492] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: Bootstrap starting.
I20260925 14:52:40.489544 23054 tablet_bootstrap.cc:654] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:40.490311 23056 tablet_bootstrap.cc:492] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: Bootstrap starting.
I20260925 14:52:40.491147 23054 log.cc:824] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:40.492833 23055 tablet_bootstrap.cc:654] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:40.493495 23056 tablet_bootstrap.cc:654] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:40.493988 23055 log.cc:824] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:40.494302 23054 tablet_bootstrap.cc:492] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511: No bootstrap required, opened a new log
I20260925 14:52:40.494526 23056 log.cc:824] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:40.494616 23054 ts_tablet_manager.cc:1397] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 14:52:40.497331 23055 tablet_bootstrap.cc:492] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: No bootstrap required, opened a new log
I20260925 14:52:40.497459 23056 tablet_bootstrap.cc:492] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: No bootstrap required, opened a new log
I20260925 14:52:40.497648 23055 ts_tablet_manager.cc:1397] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 14:52:40.497710 23056 ts_tablet_manager.cc:1397] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 14:52:40.504561 23055 raft_consensus.cc:348] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } }
I20260925 14:52:40.504972 23055 raft_consensus.cc:374] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:40.505119 23055 raft_consensus.cc:729] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f0f44dcfa4463bb2c5fd4adae14847, State: Initialized, Role: FOLLOWER
I20260925 14:52:40.504964 23054 raft_consensus.cc:348] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } }
I20260925 14:52:40.505338 23054 raft_consensus.cc:374] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:40.505486 23054 raft_consensus.cc:729] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbef1eae066f4bd994c55711819e6511, State: Initialized, Role: FOLLOWER
I20260925 14:52:40.505780 23055 consensus_queue.cc:260] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } }
I20260925 14:52:40.506054 23054 consensus_queue.cc:260] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } }
I20260925 14:52:40.507998 23056 raft_consensus.cc:348] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } }
I20260925 14:52:40.508467 23056 raft_consensus.cc:374] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:40.508644 23056 raft_consensus.cc:729] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9feb7aca38094f7f85a058226e3e696c, State: Initialized, Role: FOLLOWER
I20260925 14:52:40.509164 22775 heartbeater.cc:503] Master 127.17.133.254:44395 was elected leader, sending a full tablet report...
I20260925 14:52:40.509630 22906 heartbeater.cc:503] Master 127.17.133.254:44395 was elected leader, sending a full tablet report...
I20260925 14:52:40.509760 23056 consensus_queue.cc:260] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } }
I20260925 14:52:40.510263 23055 ts_tablet_manager.cc:1428] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260925 14:52:40.510855 23036 heartbeater.cc:503] Master 127.17.133.254:44395 was elected leader, sending a full tablet report...
I20260925 14:52:40.511106 23054 ts_tablet_manager.cc:1428] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260925 14:52:40.511420 23056 ts_tablet_manager.cc:1428] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260925 14:52:40.557921 22776 tablet.cc:2369] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:40.595978 23062 raft_consensus.cc:482] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:40.596287 23062 raft_consensus.cc:504] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } }
I20260925 14:52:40.597872 23062 leader_election.cc:288] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2f0f44dcfa4463bb2c5fd4adae14847 (127.17.133.194:40967), dbef1eae066f4bd994c55711819e6511 (127.17.133.193:43955)
I20260925 14:52:40.600453 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.598427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:45285 (local address 127.17.133.193:43955)
0925 14:52:40.598599 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.598606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.598628 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:52:40.598743 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.598749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.598807 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.598904 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.599847 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.599851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.599854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.599868 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.599901 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.599912 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.600037 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.600158 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.600212 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:52:40.600453 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.598126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43955 (local address 127.17.133.195:45285)
0925 14:52:40.598462 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.598484 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:40.598605 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:40.598938 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:40.598942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:40.598959 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:40.599655 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:40.599657 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:40.599669 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:40.599703 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:40.600024 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:40.600029 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:40.600033 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:40.600153 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 14:52:40.600202 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260925 14:52:40.600817 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.598244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:47169 (local address 127.17.133.194:40967)
0925 14:52:40.598440 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.598446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.599057 (+ 611us) server_negotiation.cc:407] Connection header received
0925 14:52:40.599183 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.599189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.599245 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.599324 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.600201 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.600206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.600210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.600233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.600262 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.600272 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.600388 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.600456 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.600510 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:52:40.600996 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.598241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:40967 (local address 127.17.133.195:47169)
0925 14:52:40.598934 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.598957 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:40.599077 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:40.599373 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:40.599378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:40.599400 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:40.600004 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:40.600006 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:40.600029 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:40.600063 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:40.600357 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:40.600361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:40.600367 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:52:40.600479 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:52:40.600543 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":359,"threads_started":1}
I20260925 14:52:40.601606 22730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41159abc074a45708473401ad1f288b7" candidate_uuid: "9feb7aca38094f7f85a058226e3e696c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbef1eae066f4bd994c55711819e6511" is_pre_election: true
I20260925 14:52:40.601946 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41159abc074a45708473401ad1f288b7" candidate_uuid: "9feb7aca38094f7f85a058226e3e696c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" is_pre_election: true
I20260925 14:52:40.602070 22730 raft_consensus.cc:2454] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9feb7aca38094f7f85a058226e3e696c in term 0.
I20260925 14:52:40.602418 22861 raft_consensus.cc:2454] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9feb7aca38094f7f85a058226e3e696c in term 0.
I20260925 14:52:40.602686 22924 leader_election.cc:302] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9feb7aca38094f7f85a058226e3e696c, dbef1eae066f4bd994c55711819e6511; no voters:
I20260925 14:52:40.603019 23062 raft_consensus.cc:2781] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:40.603153 23062 raft_consensus.cc:482] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:40.603240 23062 raft_consensus.cc:3037] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:40.605243 23062 raft_consensus.cc:504] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } }
I20260925 14:52:40.605742 23062 leader_election.cc:288] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [CANDIDATE]: Term 1 election: Requested vote from peers f2f0f44dcfa4463bb2c5fd4adae14847 (127.17.133.194:40967), dbef1eae066f4bd994c55711819e6511 (127.17.133.193:43955)
I20260925 14:52:40.606051 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41159abc074a45708473401ad1f288b7" candidate_uuid: "9feb7aca38094f7f85a058226e3e696c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847"
I20260925 14:52:40.606222 22730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41159abc074a45708473401ad1f288b7" candidate_uuid: "9feb7aca38094f7f85a058226e3e696c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbef1eae066f4bd994c55711819e6511"
I20260925 14:52:40.606253 22861 raft_consensus.cc:3037] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:40.606397 22730 raft_consensus.cc:3037] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:40.608316 22861 raft_consensus.cc:2454] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9feb7aca38094f7f85a058226e3e696c in term 1.
I20260925 14:52:40.608402 22730 raft_consensus.cc:2454] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9feb7aca38094f7f85a058226e3e696c in term 1.
I20260925 14:52:40.608675 22927 leader_election.cc:302] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9feb7aca38094f7f85a058226e3e696c, f2f0f44dcfa4463bb2c5fd4adae14847; no voters:
I20260925 14:52:40.608981 23062 raft_consensus.cc:2781] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:40.609361 23062 raft_consensus.cc:686] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 1 LEADER]: Becoming Leader. State: Replica: 9feb7aca38094f7f85a058226e3e696c, State: Running, Role: LEADER
I20260925 14:52:40.609827 23062 consensus_queue.cc:237] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } }
I20260925 14:52:40.614326 22588 catalog_manager.cc:5495] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9feb7aca38094f7f85a058226e3e696c (127.17.133.195). New cstate: current_term: 1 leader_uuid: "9feb7aca38094f7f85a058226e3e696c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9feb7aca38094f7f85a058226e3e696c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46677 } health_report { overall_health: HEALTHY } } }
I20260925 14:52:40.626755 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.624685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.17.133.195:46677)
0925 14:52:40.624853 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.624858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.624875 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:52:40.625077 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.625080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.625147 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.625250 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.626181 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.626186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.626191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.626208 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.626248 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.626258 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.626377 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.626461 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.626515 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:52:40.634663 22861 raft_consensus.cc:1261] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [term 1 FOLLOWER]: Refusing update from remote peer 9feb7aca38094f7f85a058226e3e696c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:40.634740 22730 raft_consensus.cc:1261] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [term 1 FOLLOWER]: Refusing update from remote peer 9feb7aca38094f7f85a058226e3e696c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:40.635600 23064 consensus_queue.cc:1035] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2f0f44dcfa4463bb2c5fd4adae14847" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 40967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:40.635942 23062 consensus_queue.cc:1035] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbef1eae066f4bd994c55711819e6511" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:40.649649 23067 mvcc.cc:205] Tried to move back new op lower bound from 7333265246749147136 to 7333265246661238784. Current Snapshot: MvccSnapshot[applied={T|T < 7333265246749147136}]
I20260925 14:52:40.661638 22588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:40.661911 22588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:40.665179 22588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83044b0cda16414d83e58177f1a0ee0a: Sending DeleteTablet for 3 replicas of tablet 41159abc074a45708473401ad1f288b7
I20260925 14:52:40.666203 22841 tablet_service.cc:1514] Processing DeleteTablet for tablet 41159abc074a45708473401ad1f288b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:40 UTC) from {username='slave'} at 127.0.0.1:52714
I20260925 14:52:40.666451 22710 tablet_service.cc:1514] Processing DeleteTablet for tablet 41159abc074a45708473401ad1f288b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:40 UTC) from {username='slave'} at 127.0.0.1:38422
I20260925 14:52:40.667097 23076 tablet_replica.cc:330] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: stopping tablet replica
I20260925 14:52:40.667497 23076 raft_consensus.cc:2229] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:40.667475 22971 tablet_service.cc:1514] Processing DeleteTablet for tablet 41159abc074a45708473401ad1f288b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:40 UTC) from {username='slave'} at 127.0.0.1:55682
I20260925 14:52:40.667977 23076 raft_consensus.cc:2258] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:40.668499 23078 tablet_replica.cc:330] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: stopping tablet replica
I20260925 14:52:40.668943 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 22648
I20260925 14:52:40.668921 23078 raft_consensus.cc:2229] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:40.669572 23078 raft_consensus.cc:2258] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:40.670051 23076 ts_tablet_manager.cc:1905] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:40.673944 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.671593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.17.133.193:43955)
0925 14:52:40.671756 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.671762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.671779 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:52:40.671845 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.671849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.671937 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.672032 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.673188 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.673194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.673198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.673216 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.673251 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.673266 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.673370 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.673594 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.673651 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:52:40.674888 22770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:40.677806 23078 ts_tablet_manager.cc:1905] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:40.677881 23076 ts_tablet_manager.cc:1918] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:40.678068 23076 log.cc:1196] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/41159abc074a45708473401ad1f288b7
I20260925 14:52:40.678668 23076 ts_tablet_manager.cc:1939] T 41159abc074a45708473401ad1f288b7 P f2f0f44dcfa4463bb2c5fd4adae14847: Deleting consensus metadata
I20260925 14:52:40.680382 22576 catalog_manager.cc:4841] TS f2f0f44dcfa4463bb2c5fd4adae14847 (127.17.133.194:40967): tablet 41159abc074a45708473401ad1f288b7 (table test-table [id=69ba551a012a4efd9e48299c5e2c23a2]) successfully deleted
I20260925 14:52:40.688210 23078 ts_tablet_manager.cc:1918] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:40.688470 23078 log.cc:1196] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/41159abc074a45708473401ad1f288b7
I20260925 14:52:40.689136 23078 ts_tablet_manager.cc:1939] T 41159abc074a45708473401ad1f288b7 P 9feb7aca38094f7f85a058226e3e696c: Deleting consensus metadata
I20260925 14:52:40.691004 22573 catalog_manager.cc:4841] TS 9feb7aca38094f7f85a058226e3e696c (127.17.133.195:46677): tablet 41159abc074a45708473401ad1f288b7 (table test-table [id=69ba551a012a4efd9e48299c5e2c23a2]) successfully deleted
I20260925 14:52:40.768607 23077 tablet_replica.cc:330] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511: stopping tablet replica
I20260925 14:52:40.769109 23077 raft_consensus.cc:2229] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:40.769635 23077 raft_consensus.cc:2258] T 41159abc074a45708473401ad1f288b7 P dbef1eae066f4bd994c55711819e6511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:40.771085 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 22648
W20260925 14:52:40.787952 22573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.133.193:43955 (error 108)
I20260925 14:52:40.788070 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 22779
I20260925 14:52:40.790999 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.790513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43955 (local address 127.0.0.1:38440)
0925 14:52:40.790697 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.790822 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 14:52:40.791333 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.788908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.17.133.194:40967)
0925 14:52:40.789136 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.789145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.789171 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:52:40.789351 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.789358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.789482 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.789588 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.790636 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.790642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.790646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.790671 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.790711 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.790723 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.790798 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.790989 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.791046 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 14:52:40.791633 22573 catalog_manager.cc:4584] TS dbef1eae066f4bd994c55711819e6511 (127.17.133.193:43955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41159abc074a45708473401ad1f288b7: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43955: connect: Connection refused (error 111)
I20260925 14:52:40.791973 22901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:40.845453 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.844925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43955 (local address 127.0.0.1:38450)
0925 14:52:40.845172 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.845276 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 14:52:40.846544 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.846153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43955 (local address 127.0.0.1:38454)
0925 14:52:40.846312 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.846403 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 14:52:40.878854 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 22779
I20260925 14:52:40.895267 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 22909
I20260925 14:52:40.898171 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.896026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.17.133.195:46677)
0925 14:52:40.896201 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:40.896208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:40.896234 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:52:40.896482 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:40.896486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:40.896537 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:40.896687 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:40.897554 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:40.897560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:40.897566 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:40.897587 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:40.897614 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:40.897626 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:40.897765 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:40.897821 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:52:40.897883 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:52:40.898818 23031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:40.906980 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.906593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43955 (local address 127.0.0.1:38462)
0925 14:52:40.906789 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.906864 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 14:52:40.908023 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.907730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43955 (local address 127.0.0.1:38476)
0925 14:52:40.907847 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.907908 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:52:40.988682 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.988179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43955 (local address 127.0.0.1:38486)
0925 14:52:40.988380 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.988469 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 14:52:40.989663 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.989314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43955 (local address 127.0.0.1:38500)
0925 14:52:40.989489 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.989558 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 14:52:41.015591 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 22909
I20260925 14:52:41.034212 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 22558
I20260925 14:52:41.035353 22619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:41.134506 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 22558
2026-09-25T14:52:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2536 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 14:52:41.170892 17943 mini_oidc.cc:152] Starting JWKS server
I20260925 14:52:41.173410 17943 webserver.cc:460] Webserver started at https://127.0.0.1:41143/ using document root <none> and password file <none>
I20260925 14:52:41.173712 17943 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:52:41.174345 17943 webserver.cc:460] Webserver started at http://127.0.0.1:37477/ using document root <none> and password file <none>
2026-09-25T14:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:41Z Disabled control of system clock
I20260925 14:52:41.199290 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:43151
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:42169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:43151
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41143/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/testchainca.cert with env {}
W20260925 14:52:41.505476 23090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:41.505739 23090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:41.505815 23090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:41.512519 23090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:41.512617 23090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:41.512655 23090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:41.512707 23090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:41.512748 23090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:52:41.512804 23090 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:52:41.512842 23090 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41143/jwks.json
I20260925 14:52:41.522058 23090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:42169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:43151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:43151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41143/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:41.523167 23090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:41.524282 23090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:41.535290 23096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:41.535379 23098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:41.535419 23095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:41.536092 23090 server_base.cc:1034] running on GCE node
I20260925 14:52:41.536870 23090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:41.538120 23090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:41.539328 23090 hybrid_clock.cc:648] HybridClock initialized: now 1790347961539298 us; error 44 us; skew 500 ppm
I20260925 14:52:41.539654 23090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:52:41.541782 23090 webserver.cc:460] Webserver started at http://127.17.133.254:46503/ using document root <none> and password file <none>
I20260925 14:52:41.542444 23090 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:41.542563 23090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:41.542827 23090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:41.545389 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "bc5e366be07e4f9a93bcc2941411fc5a"
format_stamp: "Formatted at 2026-09-25 14:52:41 on dist-test-slave-v5vl"
server_key: "7c3f9293a0a88188c3e9398c8f6fb07c"
server_key_iv: "f554abca672a200fbf1897d4303a4c3e"
server_key_version: "encryptionkey@0"
I20260925 14:52:41.546092 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "bc5e366be07e4f9a93bcc2941411fc5a"
format_stamp: "Formatted at 2026-09-25 14:52:41 on dist-test-slave-v5vl"
server_key: "7c3f9293a0a88188c3e9398c8f6fb07c"
server_key_iv: "f554abca672a200fbf1897d4303a4c3e"
server_key_version: "encryptionkey@0"
I20260925 14:52:41.550410 23090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:52:41.553463 23104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:41.554560 23090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 14:52:41.554735 23090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "bc5e366be07e4f9a93bcc2941411fc5a"
format_stamp: "Formatted at 2026-09-25 14:52:41 on dist-test-slave-v5vl"
server_key: "7c3f9293a0a88188c3e9398c8f6fb07c"
server_key_iv: "f554abca672a200fbf1897d4303a4c3e"
server_key_version: "encryptionkey@0"
I20260925 14:52:41.554944 23090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:41.568552 23090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:41.569269 23090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:41.569564 23090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:41.586730 23090 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:43151
I20260925 14:52:41.586748 23155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:43151 every 8 connection(s)
I20260925 14:52:41.587988 23090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:41.591697 23156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:41.596004 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 23090
I20260925 14:52:41.596196 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:41.596870 17943 external_mini_cluster.cc:1428] Setting key 5615b8b98a82aba2e9c313a6a5459a56
I20260925 14:52:41.603117 23156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a: Bootstrap starting.
I20260925 14:52:41.606505 23156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:41.607800 23156 log.cc:824] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:41.611152 23156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a: No bootstrap required, opened a new log
I20260925 14:52:41.611948 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:41.598628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.17.133.254:43151)
0925 14:52:41.599731 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:41.599756 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:41.599809 (+ 53us) server_negotiation.cc:407] Connection header received
0925 14:52:41.602177 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:41.602204 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:41.602750 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:41.603192 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:41.603677 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:41.605591 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:41.606870 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:41.607924 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:41.608101 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:41.609332 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:41.609343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:41.609358 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:41.609971 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:41.610128 (+ 157us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:41.610154 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:41.610267 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:41.610446 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 14:52:41.610742 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":362,"threads_started":1}
I20260925 14:52:41.615545 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:52:41.619347 23156 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5e366be07e4f9a93bcc2941411fc5a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43151 } }
I20260925 14:52:41.619688 23156 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:41.619787 23156 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5e366be07e4f9a93bcc2941411fc5a, State: Initialized, Role: FOLLOWER
I20260925 14:52:41.620256 23156 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5e366be07e4f9a93bcc2941411fc5a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43151 } }
I20260925 14:52:41.620505 23156 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:41.620661 23156 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:41.620849 23156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:41.622797 23156 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5e366be07e4f9a93bcc2941411fc5a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43151 } }
I20260925 14:52:41.623242 23156 leader_election.cc:302] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc5e366be07e4f9a93bcc2941411fc5a; no voters:
I20260925 14:52:41.623767 23156 leader_election.cc:288] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:41.623935 23161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:41.625618 23156 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:41.626048 23161 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [term 1 LEADER]: Becoming Leader. State: Replica: bc5e366be07e4f9a93bcc2941411fc5a, State: Running, Role: LEADER
I20260925 14:52:41.626570 23161 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5e366be07e4f9a93bcc2941411fc5a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43151 } }
I20260925 14:52:41.629305 23162 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc5e366be07e4f9a93bcc2941411fc5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5e366be07e4f9a93bcc2941411fc5a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43151 } } }
I20260925 14:52:41.629719 23162 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:41.629675 23163 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc5e366be07e4f9a93bcc2941411fc5a. Latest consensus state: current_term: 1 leader_uuid: "bc5e366be07e4f9a93bcc2941411fc5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5e366be07e4f9a93bcc2941411fc5a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43151 } } }
I20260925 14:52:41.630002 23163 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:41.631373 23172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:41.636168 23172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:41.643807 23172 catalog_manager.cc:1347] Generated new cluster ID: 94712a4cc9364168a9d59ae18fff3f87
I20260925 14:52:41.643908 23172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:41.662235 23172 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:41.663448 23172 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:41.671352 23172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc5e366be07e4f9a93bcc2941411fc5a: Generated new TSK 0
I20260925 14:52:41.672191 23172 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:52:41.723168 23186 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 14:52:41.723344 23186 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:52:41.724234 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:41.696940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.17.133.254:43151)
0925 14:52:41.697136 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:41.697145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:41.697591 (+ 446us) server_negotiation.cc:407] Connection header received
0925 14:52:41.697906 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:41.697913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:41.697983 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:41.698127 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:52:41.698622 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:41.699370 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:41.700838 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:41.701291 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:41.701436 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:41.701915 (+ 479us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:52:41.723552 (+ 21637us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:52:41.723930 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 14:52:41.724368 23159 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 14:52:41.726066 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:41.696885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:43151 (local address 127.0.0.1:53628)
0925 14:52:41.697480 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:41.697494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:41.697681 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:41.698155 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:41.698160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:41.698174 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:52:41.698472 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:41.698480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:41.699506 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:41.699512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:41.700690 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:41.700713 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:41.701462 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:41.701468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:41.701618 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:41.701669 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:52:41.723849 (+ 22180us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:52:41.724209 (+ 360us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.133.254:43151: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":319,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.17.133.254:43151: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 14:52:41.727465 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 23090
I20260925 14:52:41.728613 23151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:41.817991 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 23090
2026-09-25T14:52:41Z chronyd exiting
I20260925 14:52:41.867486 17943 test_util.cc:175] -----------------------------------------------
I20260925 14:52:41.867645 17943 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347918228911-17943-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (700 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 14:52:41.871142 17943 mini_oidc.cc:152] Starting JWKS server
I20260925 14:52:41.872932 17943 webserver.cc:460] Webserver started at https://127.0.0.1:36981/ using document root <none> and password file <none>
I20260925 14:52:41.873190 17943 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:52:41.873811 17943 webserver.cc:460] Webserver started at http://127.0.0.1:34555/ using document root <none> and password file <none>
2026-09-25T14:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:41Z Disabled control of system clock
I20260925 14:52:41.898981 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34279
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:34279
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36981/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/testchainca.cert with env {}
W20260925 14:52:42.209775 23193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:42.210042 23193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:42.210127 23193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:42.217276 23193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:42.217378 23193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:42.217414 23193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:42.217468 23193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:42.217511 23193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:52:42.217571 23193 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:52:42.217607 23193 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36981/jwks.json
I20260925 14:52:42.227041 23193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:34279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36981/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:42.228219 23193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:42.229475 23193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:42.240861 23201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:42.240891 23198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:42.240932 23199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:42.241856 23193 server_base.cc:1034] running on GCE node
I20260925 14:52:42.242640 23193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:42.244067 23193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:42.245303 23193 hybrid_clock.cc:648] HybridClock initialized: now 1790347962245270 us; error 56 us; skew 500 ppm
I20260925 14:52:42.245664 23193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:52:42.247965 23193 webserver.cc:460] Webserver started at http://127.17.133.254:42993/ using document root <none> and password file <none>
I20260925 14:52:42.248721 23193 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:42.248852 23193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:42.249185 23193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:42.251958 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "0b11300906c44c4db19d3157c208c5f7"
format_stamp: "Formatted at 2026-09-25 14:52:42 on dist-test-slave-v5vl"
server_key: "c4c13409a6019abb0f917e5a150df104"
server_key_iv: "4ba3063577e542261fa5e92a31f349d6"
server_key_version: "encryptionkey@0"
I20260925 14:52:42.252797 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "0b11300906c44c4db19d3157c208c5f7"
format_stamp: "Formatted at 2026-09-25 14:52:42 on dist-test-slave-v5vl"
server_key: "c4c13409a6019abb0f917e5a150df104"
server_key_iv: "4ba3063577e542261fa5e92a31f349d6"
server_key_version: "encryptionkey@0"
I20260925 14:52:42.257766 23193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 14:52:42.261122 23207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:42.262355 23193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:52:42.262539 23193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "0b11300906c44c4db19d3157c208c5f7"
format_stamp: "Formatted at 2026-09-25 14:52:42 on dist-test-slave-v5vl"
server_key: "c4c13409a6019abb0f917e5a150df104"
server_key_iv: "4ba3063577e542261fa5e92a31f349d6"
server_key_version: "encryptionkey@0"
I20260925 14:52:42.262758 23193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:42.276889 23193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:42.277630 23193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:42.277940 23193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:42.295879 23193 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:34279
I20260925 14:52:42.295900 23258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:34279 every 8 connection(s)
I20260925 14:52:42.297344 23193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:42.300981 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:42.307274 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 23193
I20260925 14:52:42.307512 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:42.308110 17943 external_mini_cluster.cc:1428] Setting key eeeb1e238c2bb09125bb54703f27db2e
I20260925 14:52:42.311088 23259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7: Bootstrap starting.
I20260925 14:52:42.314370 23259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:42.315428 23259 log.cc:824] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:42.318414 23259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7: No bootstrap required, opened a new log
I20260925 14:52:42.321336 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:42.310014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.17.133.254:34279)
0925 14:52:42.310966 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:42.310976 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:42.311016 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:52:42.312771 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:42.312793 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:42.313248 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:42.313554 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:42.314116 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:42.315467 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:42.317125 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:42.317751 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:42.317888 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:42.319102 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:42.319108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:42.319121 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:42.319620 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:42.319691 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:42.319709 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:42.319794 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:42.320015 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:52:42.320333 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":337,"threads_started":1}
I20260925 14:52:42.324883 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:52:42.326355 23259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b11300906c44c4db19d3157c208c5f7" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34279 } }
I20260925 14:52:42.326663 23259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:42.326792 23259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b11300906c44c4db19d3157c208c5f7, State: Initialized, Role: FOLLOWER
I20260925 14:52:42.327315 23259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b11300906c44c4db19d3157c208c5f7" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34279 } }
I20260925 14:52:42.327523 23259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:42.327666 23259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:42.327824 23259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:42.329727 23259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b11300906c44c4db19d3157c208c5f7" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34279 } }
I20260925 14:52:42.330163 23259 leader_election.cc:302] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b11300906c44c4db19d3157c208c5f7; no voters:
I20260925 14:52:42.330758 23259 leader_election.cc:288] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:42.330857 23264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:42.331688 23264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [term 1 LEADER]: Becoming Leader. State: Replica: 0b11300906c44c4db19d3157c208c5f7, State: Running, Role: LEADER
I20260925 14:52:42.332191 23264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b11300906c44c4db19d3157c208c5f7" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34279 } }
I20260925 14:52:42.332940 23259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:42.334777 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b11300906c44c4db19d3157c208c5f7. Latest consensus state: current_term: 1 leader_uuid: "0b11300906c44c4db19d3157c208c5f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b11300906c44c4db19d3157c208c5f7" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34279 } } }
I20260925 14:52:42.335091 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:42.336793 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b11300906c44c4db19d3157c208c5f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b11300906c44c4db19d3157c208c5f7" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34279 } } }
I20260925 14:52:42.337018 23269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:42.337040 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:42.343005 23269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:42.354238 23269 catalog_manager.cc:1347] Generated new cluster ID: 549d280dc6d449c1b559ce14670b0184
I20260925 14:52:42.354336 23269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:42.367527 23269 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:42.369206 23269 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:42.379866 23269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b11300906c44c4db19d3157c208c5f7: Generated new TSK 0
I20260925 14:52:42.380817 23269 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:52:42.418716 23289 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 14:52:42.418844 23289 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:52:42.420017 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:42.400412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.17.133.254:34279)
0925 14:52:42.400627 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:42.400633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:42.401164 (+ 531us) server_negotiation.cc:407] Connection header received
0925 14:52:42.401353 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:42.401359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:42.401446 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:42.401612 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:52:42.402018 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:42.403169 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:42.404529 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:42.405229 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:42.405353 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:42.405802 (+ 449us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:52:42.419363 (+ 13561us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:52:42.419723 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 14:52:42.420127 23288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:42.400422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34279 (local address 127.0.0.1:50890)
0925 14:52:42.401065 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:42.401081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:42.401193 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:42.401616 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:42.401619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:42.401628 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:52:42.401870 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:42.401877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:42.403385 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:42.403388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:42.404400 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:42.404409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:42.405379 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:42.405383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:42.405507 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:42.405535 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:52:42.419547 (+ 14012us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:52:42.419733 (+ 186us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.133.254:34279: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":364,"threads_started":1}
W20260925 14:52:42.420130 23262 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 14:52:42.421998 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 23193
I20260925 14:52:42.423131 23254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:42.513669 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 23193
2026-09-25T14:52:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (696 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 14:52:42.568097 17943 mini_oidc.cc:152] Starting JWKS server
I20260925 14:52:42.570019 17943 webserver.cc:460] Webserver started at https://127.0.0.1:40487/ using document root <none> and password file <none>
I20260925 14:52:42.570194 17943 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:52:42.570750 17943 webserver.cc:460] Webserver started at http://127.0.0.1:41631/ using document root <none> and password file <none>
2026-09-25T14:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:42Z Disabled control of system clock
I20260925 14:52:42.595083 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:42467
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:37585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:42467
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40487/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:52:42.895522 23296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:42.895768 23296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:42.895915 23296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:42.902720 23296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:42.902804 23296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:42.902927 23296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:42.903028 23296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:42.903120 23296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:52:42.903215 23296 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:52:42.903297 23296 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40487/jwks.json
I20260925 14:52:42.911451 23296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:37585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:42467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:42467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40487/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:42.912599 23296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:42.914227 23296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:42.925590 23301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:42.925654 23302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:42.925659 23304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:42.926450 23296 server_base.cc:1034] running on GCE node
I20260925 14:52:42.927215 23296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:42.928494 23296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:42.929694 23296 hybrid_clock.cc:648] HybridClock initialized: now 1790347962929664 us; error 31 us; skew 500 ppm
I20260925 14:52:42.929991 23296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:52:42.937868 23296 webserver.cc:460] Webserver started at http://127.17.133.254:35701/ using document root <none> and password file <none>
I20260925 14:52:42.938520 23296 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:42.938617 23296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:42.938887 23296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:42.941802 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "8b8f86c3d539439ab066eab4bf4fabcc"
format_stamp: "Formatted at 2026-09-25 14:52:42 on dist-test-slave-v5vl"
server_key: "84c1ef129ee7c339ca13ba451e1c6391"
server_key_iv: "f74b915f3ad3911e99d6e8b53a5c2b10"
server_key_version: "encryptionkey@0"
I20260925 14:52:42.942536 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "8b8f86c3d539439ab066eab4bf4fabcc"
format_stamp: "Formatted at 2026-09-25 14:52:42 on dist-test-slave-v5vl"
server_key: "84c1ef129ee7c339ca13ba451e1c6391"
server_key_iv: "f74b915f3ad3911e99d6e8b53a5c2b10"
server_key_version: "encryptionkey@0"
I20260925 14:52:42.947333 23296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:52:42.950577 23310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:42.951815 23296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:52:42.951941 23296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "8b8f86c3d539439ab066eab4bf4fabcc"
format_stamp: "Formatted at 2026-09-25 14:52:42 on dist-test-slave-v5vl"
server_key: "84c1ef129ee7c339ca13ba451e1c6391"
server_key_iv: "f74b915f3ad3911e99d6e8b53a5c2b10"
server_key_version: "encryptionkey@0"
I20260925 14:52:42.952121 23296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:42.968000 23296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:42.968760 23296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:42.969007 23296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:42.986801 23296 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:42467
I20260925 14:52:42.986819 23361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:42467 every 8 connection(s)
I20260925 14:52:42.988034 23296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:42.990506 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 23296
I20260925 14:52:42.990672 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:42.991307 17943 external_mini_cluster.cc:1428] Setting key aeebc538b4cde913e039906f343649bb
I20260925 14:52:42.991607 23362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:43.004678 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:42.993104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.17.133.254:42467)
0925 14:52:42.994052 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:42.994064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:42.994109 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:52:42.995778 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:42.995798 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:42.996227 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:42.996540 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:42.997151 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:42.998514 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.000309 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.001015 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.001170 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:43.002425 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:43.002432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:43.002449 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:43.002995 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:43.003069 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:43.003088 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:43.003173 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:43.003390 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 14:52:43.003723 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":356,"threads_started":1}
I20260925 14:52:43.004848 23362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc: Bootstrap starting.
I20260925 14:52:43.008196 23362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:43.008323 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:52:43.009249 23362 log.cc:824] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:43.012073 23362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc: No bootstrap required, opened a new log
I20260925 14:52:43.019084 23362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f86c3d539439ab066eab4bf4fabcc" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42467 } }
I20260925 14:52:43.019536 23362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:43.019646 23362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8f86c3d539439ab066eab4bf4fabcc, State: Initialized, Role: FOLLOWER
I20260925 14:52:43.020242 23362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f86c3d539439ab066eab4bf4fabcc" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42467 } }
I20260925 14:52:43.020454 23362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:43.020587 23362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:43.020792 23362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:43.022768 23362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f86c3d539439ab066eab4bf4fabcc" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42467 } }
I20260925 14:52:43.023237 23362 leader_election.cc:302] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8f86c3d539439ab066eab4bf4fabcc; no voters:
I20260925 14:52:43.023782 23362 leader_election.cc:288] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:43.023934 23367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:43.024597 23367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [term 1 LEADER]: Becoming Leader. State: Replica: 8b8f86c3d539439ab066eab4bf4fabcc, State: Running, Role: LEADER
I20260925 14:52:43.025122 23367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f86c3d539439ab066eab4bf4fabcc" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42467 } }
I20260925 14:52:43.025980 23362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:43.027886 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8f86c3d539439ab066eab4bf4fabcc. Latest consensus state: current_term: 1 leader_uuid: "8b8f86c3d539439ab066eab4bf4fabcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f86c3d539439ab066eab4bf4fabcc" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42467 } } }
I20260925 14:52:43.027864 23368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b8f86c3d539439ab066eab4bf4fabcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f86c3d539439ab066eab4bf4fabcc" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42467 } } }
I20260925 14:52:43.028275 23368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:43.028319 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:43.030136 23372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:43.035154 23372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:43.043643 23372 catalog_manager.cc:1347] Generated new cluster ID: 9a9e54367abe4435b2959782043a4c9c
I20260925 14:52:43.043747 23372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:43.089609 23372 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:43.091472 23372 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:43.102500 23372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b8f86c3d539439ab066eab4bf4fabcc: Generated new TSK 0
I20260925 14:52:43.103355 23372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:52:43.125209 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:43.117504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.17.133.254:42467)
0925 14:52:43.117756 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:43.117764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:43.118072 (+ 308us) server_negotiation.cc:407] Connection header received
0925 14:52:43.118317 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:43.118321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:43.118392 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:43.118558 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:52:43.119023 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.120174 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.121599 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.122087 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.122232 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:43.122721 (+ 489us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:52:43.124527 (+ 1806us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:52:43.124905 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 14:52:43.125340 23365 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 14:52:43.125792 23391 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:43.117418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42467 (local address 127.0.0.1:34586)
0925 14:52:43.117961 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:43.117976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:43.118147 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:43.118562 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:43.118567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:43.118579 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:52:43.118876 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:43.118884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.120316 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.120321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:43.121450 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:43.121461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.122286 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.122293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:43.122431 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:43.122461 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:52:43.124767 (+ 2306us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:52:43.124986 (+ 219us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.133.254:42467: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":295,"threads_started":1}
I20260925 14:52:43.128768 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 23296
I20260925 14:52:43.129722 23357 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 14:52:43.138376 23392 webserver.cc:533] Webserver: sslize failed:
E20260925 14:52:43.138460 23392 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:52:43.222350 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 23296
2026-09-25T14:52:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (708 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T14:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:43Z Disabled control of system clock
I20260925 14:52:43.301558 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37605
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:37605
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/testchainca.cert with env {}
W20260925 14:52:43.607481 23397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:43.607774 23397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:43.607944 23397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:43.614934 23397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:43.615041 23397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:43.615145 23397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:43.615247 23397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:43.615317 23397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:52:43.615401 23397 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:52:43.615451 23397 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 14:52:43.624629 23397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:37605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:43.625727 23397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:43.626946 23397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:43.638062 23402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:43.638098 23403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:43.638151 23405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:43.638907 23397 server_base.cc:1034] running on GCE node
I20260925 14:52:43.639607 23397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:43.640975 23397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:43.642191 23397 hybrid_clock.cc:648] HybridClock initialized: now 1790347963642154 us; error 32 us; skew 500 ppm
I20260925 14:52:43.642549 23397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:52:43.644719 23397 webserver.cc:460] Webserver started at http://127.17.133.254:42973/ using document root <none> and password file <none>
I20260925 14:52:43.645401 23397 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:43.645533 23397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:43.645823 23397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:43.648542 23397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "9a0e65b80a9745df9a5db4cbff00ac10"
format_stamp: "Formatted at 2026-09-25 14:52:43 on dist-test-slave-v5vl"
server_key: "93055171f2432dcded0ee780839a8ff3"
server_key_iv: "6f5617d068788e7a4b102ba4020e44c2"
server_key_version: "encryptionkey@0"
I20260925 14:52:43.649405 23397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "9a0e65b80a9745df9a5db4cbff00ac10"
format_stamp: "Formatted at 2026-09-25 14:52:43 on dist-test-slave-v5vl"
server_key: "93055171f2432dcded0ee780839a8ff3"
server_key_iv: "6f5617d068788e7a4b102ba4020e44c2"
server_key_version: "encryptionkey@0"
I20260925 14:52:43.654281 23397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:52:43.657547 23411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:43.658730 23397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 14:52:43.658910 23397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "9a0e65b80a9745df9a5db4cbff00ac10"
format_stamp: "Formatted at 2026-09-25 14:52:43 on dist-test-slave-v5vl"
server_key: "93055171f2432dcded0ee780839a8ff3"
server_key_iv: "6f5617d068788e7a4b102ba4020e44c2"
server_key_version: "encryptionkey@0"
I20260925 14:52:43.659116 23397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:43.673249 23397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:43.674012 23397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:43.674281 23397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:43.691943 23397 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:37605
I20260925 14:52:43.691962 23462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:37605 every 8 connection(s)
I20260925 14:52:43.693298 23397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:43.696791 23463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:43.697453 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 23397
I20260925 14:52:43.697672 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:43.698236 17943 external_mini_cluster.cc:1428] Setting key b92f7b5bd86907e7c724cdaaa9b0a5d9
I20260925 14:52:43.710656 23463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10: Bootstrap starting.
I20260925 14:52:43.712749 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:43.700116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.17.133.254:37605)
0925 14:52:43.701460 (+ 1344us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:43.701473 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:43.701524 (+ 51us) server_negotiation.cc:407] Connection header received
0925 14:52:43.703354 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:43.703380 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:43.703944 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:43.704350 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:43.704900 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.706771 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.707969 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.708993 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.709148 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:43.710123 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:43.710133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:43.710148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:43.710788 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:43.710875 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:43.710906 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:43.711011 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:43.711169 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:52:43.711470 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":682,"threads_started":1}
I20260925 14:52:43.714361 23463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:43.715523 23463 log.cc:824] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:43.716351 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:52:43.718597 23463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10: No bootstrap required, opened a new log
I20260925 14:52:43.726042 23463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0e65b80a9745df9a5db4cbff00ac10" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37605 } }
I20260925 14:52:43.726352 23463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:43.726423 23463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0e65b80a9745df9a5db4cbff00ac10, State: Initialized, Role: FOLLOWER
I20260925 14:52:43.726862 23463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0e65b80a9745df9a5db4cbff00ac10" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37605 } }
I20260925 14:52:43.727043 23463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:43.727131 23463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:43.727247 23463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:43.729228 23463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0e65b80a9745df9a5db4cbff00ac10" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37605 } }
I20260925 14:52:43.729629 23463 leader_election.cc:302] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a0e65b80a9745df9a5db4cbff00ac10; no voters:
I20260925 14:52:43.730078 23463 leader_election.cc:288] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:43.730233 23468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:43.730911 23468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [term 1 LEADER]: Becoming Leader. State: Replica: 9a0e65b80a9745df9a5db4cbff00ac10, State: Running, Role: LEADER
I20260925 14:52:43.731365 23468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0e65b80a9745df9a5db4cbff00ac10" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37605 } }
I20260925 14:52:43.732010 23463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:43.734704 23470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a0e65b80a9745df9a5db4cbff00ac10. Latest consensus state: current_term: 1 leader_uuid: "9a0e65b80a9745df9a5db4cbff00ac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0e65b80a9745df9a5db4cbff00ac10" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37605 } } }
I20260925 14:52:43.735071 23470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:43.735917 23477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:43.734815 23469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a0e65b80a9745df9a5db4cbff00ac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0e65b80a9745df9a5db4cbff00ac10" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37605 } } }
I20260925 14:52:43.736157 23469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:43.740046 23477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:43.748086 23477 catalog_manager.cc:1347] Generated new cluster ID: 55cd55a7395a47c78693509eaec2da61
I20260925 14:52:43.748179 23477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:43.762877 23477 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:43.764139 23477 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:43.777426 23477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a0e65b80a9745df9a5db4cbff00ac10: Generated new TSK 0
I20260925 14:52:43.778234 23477 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:52:43.856822 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:43.794173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.17.133.254:37605)
0925 14:52:43.794388 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:43.794397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:43.794861 (+ 464us) server_negotiation.cc:407] Connection header received
0925 14:52:43.795105 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:43.795109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:43.795173 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:43.795301 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:52:43.795790 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.796519 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.798483 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.798857 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.798942 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:43.799462 (+ 520us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:52:43.856185 (+ 56723us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:52:43.856510 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46258: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 14:52:43.856982 23466 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46258: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 14:52:43.857193 23492 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:43.794158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37605 (local address 127.0.0.1:46258)
0925 14:52:43.794753 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:43.794773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:43.794928 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:43.795338 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:43.795343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:43.795357 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:52:43.795647 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:43.795660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.796684 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.796690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:43.798348 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:43.798361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:43.799016 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:43.799023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:43.799200 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:43.799242 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:52:43.856459 (+ 57217us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:52:43.856794 (+ 335us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.133.254:37605: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":316,"threads_started":1}
I20260925 14:52:43.858260 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 23397
I20260925 14:52:43.859313 23458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:43.955410 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 23397
2026-09-25T14:52:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (712 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (402 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (391 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:44 dist-test-slave-v5vl krb5kdc[23507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:44 dist-test-slave-v5vl krb5kdc[23507](info): set up 2 sockets
Sep 25 14:52:44 dist-test-slave-v5vl krb5kdc[23507](info): commencing operation
krb5kdc: starting...
W20260925 14:52:45.809084 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:45 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:52:45.997288 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:42599
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:42599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:46.304654 23520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:46.304939 23520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:46.305101 23520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:46.312299 23520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:46.312417 23520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:46.312557 23520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:46.312690 23520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:46.312772 23520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:46.322211 23520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:42599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:42599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:46.323598 23520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:46.325008 23520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:46.336469 23528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:46.336517 23525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:46.336547 23526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:46.337339 23520 server_base.cc:1034] running on GCE node
I20260925 14:52:46.338125 23520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:46.339416 23520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:46.340694 23520 hybrid_clock.cc:648] HybridClock initialized: now 1790347966340610 us; error 24 us; skew 500 ppm
I20260925 14:52:46.340991 23520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:46 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347966, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:46.343945 23520 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:52:46.345376 23520 webserver.cc:460] Webserver started at http://127.17.133.254:38271/ using document root <none> and password file <none>
I20260925 14:52:46.346025 23520 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:46.346119 23520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:46.346369 23520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:46.349278 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "7751544c1d4343dd93511d2422bdfe2e"
format_stamp: "Formatted at 2026-09-25 14:52:46 on dist-test-slave-v5vl"
server_key: "45df7337dc57068700e1d047ec008c4a"
server_key_iv: "5861c5f0cabd2fa1133bb7a21e5e3155"
server_key_version: "encryptionkey@0"
I20260925 14:52:46.349987 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "7751544c1d4343dd93511d2422bdfe2e"
format_stamp: "Formatted at 2026-09-25 14:52:46 on dist-test-slave-v5vl"
server_key: "45df7337dc57068700e1d047ec008c4a"
server_key_iv: "5861c5f0cabd2fa1133bb7a21e5e3155"
server_key_version: "encryptionkey@0"
I20260925 14:52:46.354980 23520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:52:46.358289 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:46.359632 23520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:52:46.359779 23520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "7751544c1d4343dd93511d2422bdfe2e"
format_stamp: "Formatted at 2026-09-25 14:52:46 on dist-test-slave-v5vl"
server_key: "45df7337dc57068700e1d047ec008c4a"
server_key_iv: "5861c5f0cabd2fa1133bb7a21e5e3155"
server_key_version: "encryptionkey@0"
I20260925 14:52:46.359937 23520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:46.383791 23520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:46.387641 23520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:46.387902 23520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:46.405428 23520 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:42599
I20260925 14:52:46.405462 23586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:42599 every 8 connection(s)
I20260925 14:52:46.406697 23520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:46.410189 23587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:46.414430 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 23520
I20260925 14:52:46.414650 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:46.415242 17943 external_mini_cluster.cc:1428] Setting key 6ff5591df67d2cad2acbfa6dc62aa660
I20260925 14:52:46.421518 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e: Bootstrap starting.
I20260925 14:52:46.424930 23587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:46.425885 23587 log.cc:824] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:46.428192 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e: No bootstrap required, opened a new log
Sep 25 14:52:46 dist-test-slave-v5vl krb5kdc[23507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:46.438354 23587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751544c1d4343dd93511d2422bdfe2e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42599 } }
I20260925 14:52:46.438877 23587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:46.439008 23587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7751544c1d4343dd93511d2422bdfe2e, State: Initialized, Role: FOLLOWER
I20260925 14:52:46.439402 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:46.417132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.17.133.254:42599)
0925 14:52:46.417981 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:46.418002 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:46.418055 (+ 53us) server_negotiation.cc:407] Connection header received
0925 14:52:46.420432 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:46.420457 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:46.420952 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:46.421361 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:46.422589 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:46.424561 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:46.425793 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:46.426676 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:46.426869 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:46.430695 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:46.430706 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:46.430723 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:46.430769 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:46.434655 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:46.435296 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:46.435305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:46.435312 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:46.435502 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:46.435810 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:46.435815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:46.435818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:46.436491 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:46.436815 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:46.437855 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0925 14:52:46.438136 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":330,"threads_started":1}
I20260925 14:52:46.439651 23587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751544c1d4343dd93511d2422bdfe2e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42599 } }
I20260925 14:52:46.439975 23587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:46.440130 23587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:46.440316 23587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:46.443181 23587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751544c1d4343dd93511d2422bdfe2e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42599 } }
I20260925 14:52:46.443634 23587 leader_election.cc:302] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7751544c1d4343dd93511d2422bdfe2e; no voters:
I20260925 14:52:46.444213 23587 leader_election.cc:288] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:46.444305 23592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:46.445026 23592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [term 1 LEADER]: Becoming Leader. State: Replica: 7751544c1d4343dd93511d2422bdfe2e, State: Running, Role: LEADER
I20260925 14:52:46.445480 23592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751544c1d4343dd93511d2422bdfe2e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42599 } }
I20260925 14:52:46.446197 23587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:46.448915 23594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7751544c1d4343dd93511d2422bdfe2e. Latest consensus state: current_term: 1 leader_uuid: "7751544c1d4343dd93511d2422bdfe2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751544c1d4343dd93511d2422bdfe2e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42599 } } }
I20260925 14:52:46.449326 23594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:46.449785 23593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7751544c1d4343dd93511d2422bdfe2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751544c1d4343dd93511d2422bdfe2e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42599 } } }
I20260925 14:52:46.450078 23593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:46.453667 23602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:46.458745 23602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:46.466459 23602 catalog_manager.cc:1347] Generated new cluster ID: 5af4240399664c99970efc5cfe130410
I20260925 14:52:46.466563 23602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:46.484037 23602 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:46.485351 23602 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:46.497429 23602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7751544c1d4343dd93511d2422bdfe2e: Generated new TSK 0
I20260925 14:52:46.498148 23602 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:52:46.546345 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42599
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:46.862358 23613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:46.862664 23613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:46.862883 23613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:46.870429 23613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:46.870585 23613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:46.870759 23613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:46.880415 23613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:46.881721 23613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:46.883044 23613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:46.895601 23618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:46.895515 23619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:46.895570 23621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:46.896224 23613 server_base.cc:1034] running on GCE node
I20260925 14:52:46.896900 23613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:46.897807 23613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:46.899029 23613 hybrid_clock.cc:648] HybridClock initialized: now 1790347966898985 us; error 68 us; skew 500 ppm
I20260925 14:52:46.899323 23613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:46 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347966, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:46.902411 23613 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:52:46.903836 23613 webserver.cc:460] Webserver started at http://127.17.133.193:35315/ using document root <none> and password file <none>
I20260925 14:52:46.904465 23613 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:46.904656 23613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:46.904956 23613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:46.907665 23613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "72a9ef25592f4227b52e8ce279172464"
format_stamp: "Formatted at 2026-09-25 14:52:46 on dist-test-slave-v5vl"
server_key: "d39d8e5ff5765cfd693e31f588f40d54"
server_key_iv: "2f67f03d31764b17e0d07f7c45601dad"
server_key_version: "encryptionkey@0"
I20260925 14:52:46.908416 23613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "72a9ef25592f4227b52e8ce279172464"
format_stamp: "Formatted at 2026-09-25 14:52:46 on dist-test-slave-v5vl"
server_key: "d39d8e5ff5765cfd693e31f588f40d54"
server_key_iv: "2f67f03d31764b17e0d07f7c45601dad"
server_key_version: "encryptionkey@0"
I20260925 14:52:46.913453 23613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 14:52:46.916785 23628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:46.918004 23613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:52:46.918164 23613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "72a9ef25592f4227b52e8ce279172464"
format_stamp: "Formatted at 2026-09-25 14:52:46 on dist-test-slave-v5vl"
server_key: "d39d8e5ff5765cfd693e31f588f40d54"
server_key_iv: "2f67f03d31764b17e0d07f7c45601dad"
server_key_version: "encryptionkey@0"
I20260925 14:52:46.918362 23613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:46.942601 23613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:46.946521 23613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:46.946748 23613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:46.947657 23613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:46.948872 23613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:46.948949 23613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:46.949103 23613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:46.949196 23613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:46.981036 23613 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:34513
I20260925 14:52:46.981108 23740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:34513 every 8 connection(s)
I20260925 14:52:46.982293 23613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:46.986081 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 23613
I20260925 14:52:46.986295 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:46.986934 17943 external_mini_cluster.cc:1428] Setting key f9b7a475df5c76d743141bdfa2de277e
Sep 25 14:52:46 dist-test-slave-v5vl krb5kdc[23507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347966, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:47.002030 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:46.984758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:39859 (local address 127.17.133.254:42599)
0925 14:52:46.985199 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:46.985203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:46.986110 (+ 907us) server_negotiation.cc:407] Connection header received
0925 14:52:46.988174 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:46.988178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:46.988239 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:46.988330 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:46.989948 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:46.990672 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:46.992138 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:46.992533 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:46.992726 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:46.996398 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:46.996403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:46.996406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:46.996434 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:46.999009 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:46.999758 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:46.999766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:46.999770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:46.999889 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:47.000298 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:47.000305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:47.000308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:47.000557 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:47.000754 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:47.001468 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 14:52:47.001680 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":241,"threads_started":1}
I20260925 14:52:47.002959 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:46.985245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42599 (local address 127.17.133.193:39859)
0925 14:52:46.985965 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:46.986001 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:46.987903 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:46.988635 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:46.988646 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:46.989225 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:46.989756 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:46.989799 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:46.990812 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:46.990817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:46.991996 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:46.992006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:46.992803 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:46.992812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:46.993033 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:46.993708 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:46.993740 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:46.996199 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:46.999200 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:46.999209 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:46.999224 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:46.999646 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:46.999993 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:46.999999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:47.000003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:47.000182 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:47.000742 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:47.000749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:47.001040 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 14:52:47.001396 (+ 356us) client_negotiation.cc:240] Negotiation successful
0925 14:52:47.001853 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":322,"threads_started":1}
I20260925 14:52:47.004544 23741 heartbeater.cc:347] Connected to a master server at 127.17.133.254:42599
I20260925 14:52:47.004864 23741 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:52:47.005537 23741 heartbeater.cc:511] Master 127.17.133.254:42599 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
I20260925 14:52:47.007496 23552 ts_manager.cc:194] Registered new tserver with Master: 72a9ef25592f4227b52e8ce279172464 (127.17.133.193:34513)
I20260925 14:52:47.008908 23552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:39859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:52:47.029840 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42599
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:47.337697 23747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:47.337971 23747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:47.338160 23747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:47.346068 23747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:47.346388 23747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:47.346640 23747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:47.356797 23747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:47.358069 23747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:47.359335 23747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:47.371841 23753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:47.371906 23755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:47.372128 23752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:47.372865 23747 server_base.cc:1034] running on GCE node
I20260925 14:52:47.373456 23747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:47.374354 23747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:47.375576 23747 hybrid_clock.cc:648] HybridClock initialized: now 1790347967375537 us; error 66 us; skew 500 ppm
I20260925 14:52:47.375883 23747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:47 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347967, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:47.378913 23747 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:52:47.380344 23747 webserver.cc:460] Webserver started at http://127.17.133.194:38345/ using document root <none> and password file <none>
I20260925 14:52:47.381088 23747 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:47.381206 23747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:47.381439 23747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:47.384222 23747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "2a1900459b65472a8cb8cb7a46effa86"
format_stamp: "Formatted at 2026-09-25 14:52:47 on dist-test-slave-v5vl"
server_key: "e95b5d64af7834f77e8540c44e163e63"
server_key_iv: "114d8fc8b349b252742845e5de4832d9"
server_key_version: "encryptionkey@0"
I20260925 14:52:47.384963 23747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "2a1900459b65472a8cb8cb7a46effa86"
format_stamp: "Formatted at 2026-09-25 14:52:47 on dist-test-slave-v5vl"
server_key: "e95b5d64af7834f77e8540c44e163e63"
server_key_iv: "114d8fc8b349b252742845e5de4832d9"
server_key_version: "encryptionkey@0"
I20260925 14:52:47.389678 23747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 14:52:47.392885 23762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:47.394120 23747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:52:47.394238 23747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "2a1900459b65472a8cb8cb7a46effa86"
format_stamp: "Formatted at 2026-09-25 14:52:47 on dist-test-slave-v5vl"
server_key: "e95b5d64af7834f77e8540c44e163e63"
server_key_iv: "114d8fc8b349b252742845e5de4832d9"
server_key_version: "encryptionkey@0"
I20260925 14:52:47.394389 23747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:47.415961 23747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:47.420209 23747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:47.420475 23747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:47.421438 23747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:47.422876 23747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:47.422971 23747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:47.423103 23747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:47.423168 23747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:47.454509 23747 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:40449
I20260925 14:52:47.454540 23874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:40449 every 8 connection(s)
I20260925 14:52:47.455724 23747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:47.458109 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 23747
I20260925 14:52:47.458302 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:47.458719 17943 external_mini_cluster.cc:1428] Setting key c371774e85521edd54af6aee643c1449
Sep 25 14:52:47 dist-test-slave-v5vl krb5kdc[23507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347967, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:47.476439 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:47.458170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:60457 (local address 127.17.133.254:42599)
0925 14:52:47.458365 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:47.458371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:47.459666 (+ 1295us) server_negotiation.cc:407] Connection header received
0925 14:52:47.461681 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:47.461686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:47.461763 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:47.461880 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:47.463483 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.464270 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:47.465836 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.466538 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:47.466672 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:47.470058 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:47.470066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:47.470073 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:47.470114 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:47.473417 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:47.474038 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:47.474046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:47.474052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:47.474194 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:47.474582 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:47.474588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:47.474592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:47.474959 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:47.475116 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:47.475741 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 14:52:47.475996 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 14:52:47.476986 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:47.458681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42599 (local address 127.17.133.194:60457)
0925 14:52:47.459512 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:47.459556 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:47.461429 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:47.462112 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:47.462120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:47.462687 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:47.463305 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:47.463322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.464412 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:47.464419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:47.465675 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:47.465687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.466746 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:47.466751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:47.466937 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:47.467562 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:47.467585 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:47.469855 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:47.473532 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:47.473538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:47.473549 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:47.473910 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:47.474292 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:47.474295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:47.474297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:47.474446 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:47.475081 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:47.475087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:47.475370 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 14:52:47.475573 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 14:52:47.475917 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":369,"threads_started":1}
I20260925 14:52:47.478405 23875 heartbeater.cc:347] Connected to a master server at 127.17.133.254:42599
I20260925 14:52:47.478726 23875 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
I20260925 14:52:47.479614 23875 heartbeater.cc:511] Master 127.17.133.254:42599 requested a full tablet report, sending...
I20260925 14:52:47.481040 23552 ts_manager.cc:194] Registered new tserver with Master: 2a1900459b65472a8cb8cb7a46effa86 (127.17.133.194:40449)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
I20260925 14:52:47.481972 23552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:60457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:52:47.504904 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42599
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:47.818133 23880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:47.818423 23880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:47.818588 23880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:47.825625 23880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:47.825752 23880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:47.825932 23880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:47.835032 23880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:47.836254 23880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:47.837539 23880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:47.850006 23885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:47.850051 23888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:47.850080 23886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:47.850786 23880 server_base.cc:1034] running on GCE node
I20260925 14:52:47.851264 23880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:47.852046 23880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:47.853220 23880 hybrid_clock.cc:648] HybridClock initialized: now 1790347967853174 us; error 41 us; skew 500 ppm
I20260925 14:52:47.853518 23880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:47 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347967, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:47.856468 23880 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:52:47.857812 23880 webserver.cc:460] Webserver started at http://127.17.133.195:33969/ using document root <none> and password file <none>
I20260925 14:52:47.858479 23880 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:47.858618 23880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:47.858911 23880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:47.861716 23880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "7c0305344db2433c924448a17d794da1"
format_stamp: "Formatted at 2026-09-25 14:52:47 on dist-test-slave-v5vl"
server_key: "ccc9a75ee1fc9e4bc34feb09f8bda955"
server_key_iv: "9887bacd9603ea8d615664b9dda16df9"
server_key_version: "encryptionkey@0"
I20260925 14:52:47.862468 23880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "7c0305344db2433c924448a17d794da1"
format_stamp: "Formatted at 2026-09-25 14:52:47 on dist-test-slave-v5vl"
server_key: "ccc9a75ee1fc9e4bc34feb09f8bda955"
server_key_iv: "9887bacd9603ea8d615664b9dda16df9"
server_key_version: "encryptionkey@0"
I20260925 14:52:47.867532 23880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 14:52:47.870784 23895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:47.872144 23880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:52:47.872305 23880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "7c0305344db2433c924448a17d794da1"
format_stamp: "Formatted at 2026-09-25 14:52:47 on dist-test-slave-v5vl"
server_key: "ccc9a75ee1fc9e4bc34feb09f8bda955"
server_key_iv: "9887bacd9603ea8d615664b9dda16df9"
server_key_version: "encryptionkey@0"
I20260925 14:52:47.872510 23880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:47.890024 23880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:47.894279 23880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:47.894557 23880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:47.895437 23880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:47.896762 23880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:47.896832 23880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:47.897019 23880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:47.897084 23880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:47.929111 23880 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:40545
I20260925 14:52:47.929154 24007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:40545 every 8 connection(s)
I20260925 14:52:47.930305 23880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:47.932703 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 23880
I20260925 14:52:47.932888 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:47.933408 17943 external_mini_cluster.cc:1428] Setting key e6e38d74cbd6b461e965c123d297837f
Sep 25 14:52:47 dist-test-slave-v5vl krb5kdc[23507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347967, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:52:47.954509 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:47.932799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:60195 (local address 127.17.133.254:42599)
0925 14:52:47.932964 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:47.932969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:47.934415 (+ 1446us) server_negotiation.cc:407] Connection header received
0925 14:52:47.936706 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:47.936711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:47.936802 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:47.936905 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:47.938332 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.939044 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:47.940521 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.940967 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:47.941093 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:47.949158 (+ 8065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:47.949181 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:47.949185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:47.949214 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:47.951555 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:47.952302 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:47.952309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:47.952315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:47.952451 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:47.952878 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:47.952886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:47.952889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:47.953161 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:47.953306 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:47.953939 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 14:52:47.954157 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:52:47.955420 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:47.933435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42599 (local address 127.17.133.195:60195)
0925 14:52:47.934270 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:47.934310 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:47.936409 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:47.937127 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:47.937136 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:47.937622 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:47.938180 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:47.938197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.939199 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:47.939204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:47.940371 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:47.940381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.941171 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:47.941181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:47.941420 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:47.946691 (+ 5271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:47.946717 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:47.948959 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:47.951711 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:47.951719 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:47.951739 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:47.952185 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:47.952557 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:47.952563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:47.952567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:47.952774 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:47.953281 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:47.953288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:47.953609 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 14:52:47.953862 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 14:52:47.954366 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":342,"threads_started":1}
I20260925 14:52:47.956738 24008 heartbeater.cc:347] Connected to a master server at 127.17.133.254:42599
I20260925 14:52:47.957008 24008 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:47.957657 24008 heartbeater.cc:511] Master 127.17.133.254:42599 requested a full tablet report, sending...
I20260925 14:52:47.959024 23552 ts_manager.cc:194] Registered new tserver with Master: 7c0305344db2433c924448a17d794da1 (127.17.133.195:40545)
I20260925 14:52:47.959987 23552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:60195
I20260925 14:52:47.960748 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/test-admin.keytab.
Sep 25 14:52:47 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:47.986627 17943 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 14:52:47.991137 24015 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:47.989005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.17.133.193:34513)
0925 14:52:47.989572 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:47.989579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:47.989624 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:52:47.989678 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:47.989683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:47.989867 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:47.989988 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:47.990773 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40030: BlockingRecv error: recv got EOF from 127.0.0.1:40030 (error 108)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":314,"threads_started":1}
W20260925 14:52:47.991586 17943 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5cc)
Sep 25 14:52:47 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:47.993043 17943 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:52:47 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:47.994299 17943 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:52:47 dist-test-slave-v5vl krb5kdc[23507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:48.008302 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:47.995814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.17.133.193:34513)
0925 14:52:47.995990 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:47.995994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:47.996063 (+ 69us) server_negotiation.cc:407] Connection header received
0925 14:52:47.996170 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:47.996173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:47.996218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:47.996290 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:47.997192 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.998217 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:47.999404 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:47.999895 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.000003 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:48.003280 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:48.003290 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:48.003303 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:48.003332 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:48.005875 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.006482 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.006489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.006491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.006601 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.006941 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.006944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.006946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.007271 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:48.007483 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:48.007741 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 14:52:48.008029 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:52:48.011411 23741 heartbeater.cc:503] Master 127.17.133.254:42599 was elected leader, sending a full tablet report...
W20260925 14:52:48.014292 24015 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:48.012414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.17.133.193:34513)
0925 14:52:48.012611 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:48.012617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:48.012992 (+ 375us) server_negotiation.cc:407] Connection header received
0925 14:52:48.013165 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:48.013169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:48.013222 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:48.013296 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:48.013988 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40046: BlockingRecv error: recv got EOF from 127.0.0.1:40046 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 14:52:48.014467 17943 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:48.015885 17943 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:48.017239 17943 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[23507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:48.029958 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:48.018548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.17.133.193:34513)
0925 14:52:48.018730 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:48.018734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:48.018810 (+ 76us) server_negotiation.cc:407] Connection header received
0925 14:52:48.018929 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:48.018932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:48.018982 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:48.019055 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:48.019901 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.020641 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.021970 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.022432 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.022521 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:48.025691 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:48.025696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:48.025699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:48.025726 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:48.027936 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.028440 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.028444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.028446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.028552 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.028905 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.028908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.028910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.029145 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:48.029316 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:48.029523 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 14:52:48.029695 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 14:52:48.033838 24015 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:48.032221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.17.133.193:34513)
0925 14:52:48.032407 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:48.032412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:48.032679 (+ 267us) server_negotiation.cc:407] Connection header received
0925 14:52:48.032845 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:48.032854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:48.032913 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:48.033008 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:48.033640 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40066: BlockingRecv error: recv got EOF from 127.0.0.1:40066 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 14:52:48.034062 17943 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347918228911-17943-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:48.035441 17943 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:48.036755 17943 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[23507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:52:48.049878 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:48.037888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.17.133.193:34513)
0925 14:52:48.038096 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:48.038101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:48.038331 (+ 230us) server_negotiation.cc:407] Connection header received
0925 14:52:48.038432 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:48.038435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:48.038480 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:48.038553 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:48.039368 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.039999 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.041209 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.041578 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.041662 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:48.044844 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:48.044848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:48.044851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:48.044882 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:48.047495 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.048099 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.048106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.048114 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.048245 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.048624 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.048631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.048634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.048875 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:48.049189 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:48.049370 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 14:52:48.049550 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:52:48.051203 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 23613
I20260925 14:52:48.061872 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:48.051775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.17.133.193:34513)
0925 14:52:48.051922 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:48.051926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:48.052202 (+ 276us) server_negotiation.cc:407] Connection header received
0925 14:52:48.052312 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:48.052315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:48.052358 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:48.052429 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:48.053336 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.053995 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.055145 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.055543 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.055638 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:48.057468 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:48.057473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:48.057476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:48.057502 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:48.059843 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.060364 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.060367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.060369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.060483 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.060839 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.060842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.060844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.061074 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:48.061247 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:48.061491 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 14:52:48.061632 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:52:48.062384 23736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:48.144527 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 23613
I20260925 14:52:48.160399 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 23747
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[23507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:52:48.177881 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:48.161232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.17.133.194:40449)
0925 14:52:48.161771 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:48.161777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:48.161802 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:52:48.161870 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:48.161877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:48.162071 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:48.162200 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:48.163161 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.164224 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.165528 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.166052 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.166156 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:48.170194 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:48.170212 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:48.170234 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:48.170287 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:48.174399 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.175229 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.175238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.175242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.175388 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.175894 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.175901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.175905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.176384 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:48.176748 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:48.177192 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 14:52:48.177462 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":290,"threads_started":1}
I20260925 14:52:48.179587 23870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:48.260042 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 23747
I20260925 14:52:48.275938 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 23880
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[23507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:52:48.290549 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:48.276824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.17.133.195:40545)
0925 14:52:48.277441 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:48.277447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:48.277475 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:52:48.277536 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:48.277541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:48.277749 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:48.277875 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:48.278813 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.279807 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.281234 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:48.281801 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:48.281898 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:48.285661 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:48.285673 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:48.285686 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:48.285715 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:48.288221 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.288926 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.288933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.288935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.289041 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:48.289341 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:48.289344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:48.289346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:48.289656 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:48.289872 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:48.290096 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 14:52:48.290273 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":312,"threads_started":1}
I20260925 14:52:48.291787 24003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:48.372514 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 23880
I20260925 14:52:48.387938 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 23520
I20260925 14:52:48.389022 23582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:48.490429 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 23520
2026-09-25T14:52:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3746 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[24035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[24035](info): set up 2 sockets
Sep 25 14:52:48 dist-test-slave-v5vl krb5kdc[24035](info): commencing operation
krb5kdc: starting...
W20260925 14:52:49.565285 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:49 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.254.keytab.
Entry for principal oryx/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.254.keytab.
I20260925 14:52:49.757937 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:40045
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.254.keytab
--principal=oryx/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:40045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:50.065714 24048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:50.066015 24048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:50.066157 24048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:50.073127 24048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:50.073246 24048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:50.073338 24048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:50.073432 24048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:50.073496 24048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:50.082916 24048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:40045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.254.keytab
--principal=oryx/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:40045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:50.084183 24048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:50.085600 24048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:50.097074 24054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:50.097141 24053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:50.097118 24056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:50.097932 24048 server_base.cc:1034] running on GCE node
I20260925 14:52:50.098625 24048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:50.099886 24048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:50.101069 24048 hybrid_clock.cc:648] HybridClock initialized: now 1790347970101033 us; error 28 us; skew 500 ppm
I20260925 14:52:50.101356 24048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:50 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347970, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:50.104173 24048 init.cc:377] Logged in from keytab as oryx/127.17.133.254@KRBTEST.COM (short username oryx)
I20260925 14:52:50.105563 24048 webserver.cc:460] Webserver started at http://127.17.133.254:36297/ using document root <none> and password file <none>
I20260925 14:52:50.106204 24048 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:50.106325 24048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:50.106607 24048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:50.109479 24048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b"
format_stamp: "Formatted at 2026-09-25 14:52:50 on dist-test-slave-v5vl"
server_key: "d14eaaa07f9e21c90cace0dab1ce037b"
server_key_iv: "f456b89caf3162896173fb5c0f0193b0"
server_key_version: "encryptionkey@0"
I20260925 14:52:50.110172 24048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b"
format_stamp: "Formatted at 2026-09-25 14:52:50 on dist-test-slave-v5vl"
server_key: "d14eaaa07f9e21c90cace0dab1ce037b"
server_key_iv: "f456b89caf3162896173fb5c0f0193b0"
server_key_version: "encryptionkey@0"
I20260925 14:52:50.114917 24048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:52:50.118026 24063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:50.119248 24048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:50.119412 24048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b"
format_stamp: "Formatted at 2026-09-25 14:52:50 on dist-test-slave-v5vl"
server_key: "d14eaaa07f9e21c90cace0dab1ce037b"
server_key_iv: "f456b89caf3162896173fb5c0f0193b0"
server_key_version: "encryptionkey@0"
I20260925 14:52:50.119601 24048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:50.135205 24048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:50.138831 24048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:50.139107 24048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:50.156670 24048 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:40045
I20260925 14:52:50.156706 24114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:40045 every 8 connection(s)
I20260925 14:52:50.157948 24048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:50.161499 24115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:50.163937 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 24048
I20260925 14:52:50.164182 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:50.165017 17943 external_mini_cluster.cc:1428] Setting key fb64808a55b40be32686caf09be42951
I20260925 14:52:50.174696 24115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: Bootstrap starting.
I20260925 14:52:50.178735 24115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:50.180001 24115 log.cc:824] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:50.182628 24115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: No bootstrap required, opened a new log
Sep 25 14:52:50 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:52:50.192517 24115 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } }
I20260925 14:52:50.193006 24115 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:50.193153 24115 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ddf08cf4ea40398ad8a455ebdd9f6b, State: Initialized, Role: FOLLOWER
I20260925 14:52:50.193835 24115 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } }
I20260925 14:52:50.194078 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:50.166950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.17.133.254:40045)
0925 14:52:50.167699 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:50.167709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:50.167744 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:52:50.169722 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:50.169751 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:50.170250 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:50.170696 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:50.173216 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:50.175033 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:50.176831 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:50.177918 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:50.178100 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:50.185530 (+ 7430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:50.185542 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:50.185570 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:50.185627 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:50.189648 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:50.190312 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:50.190323 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:50.190336 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:50.190557 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:50.190884 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:50.190890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:50.190894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:50.191608 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:50.191903 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:50.192199 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 14:52:50.192665 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":356,"threads_started":1}
I20260925 14:52:50.194120 24115 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:50.194267 24115 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:50.194443 24115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:50.197261 24115 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } }
I20260925 14:52:50.197665 24115 leader_election.cc:302] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1ddf08cf4ea40398ad8a455ebdd9f6b; no voters:
I20260925 14:52:50.198160 24115 leader_election.cc:288] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:50.198304 24120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:50.198982 24120 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 1 LEADER]: Becoming Leader. State: Replica: c1ddf08cf4ea40398ad8a455ebdd9f6b, State: Running, Role: LEADER
I20260925 14:52:50.199473 24120 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } }
I20260925 14:52:50.200289 24115 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:50.202360 24121 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } }
I20260925 14:52:50.202766 24121 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:50.202385 24122 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ddf08cf4ea40398ad8a455ebdd9f6b. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } }
I20260925 14:52:50.203090 24122 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:50.205318 24126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:50.210961 24126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:50.219137 24126 catalog_manager.cc:1347] Generated new cluster ID: f6ee1cb7e85e4426a3353ff2e9df5f63
I20260925 14:52:50.219240 24126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:50.258471 24126 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:50.260375 24126 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:50.268527 24126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: Generated new TSK 0
I20260925 14:52:50.269479 24126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab.
Entry for principal oryx/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab.
I20260925 14:52:50.332242 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40045
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab
--principal=oryx/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:50.646967 24141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:50.647240 24141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:50.647414 24141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:50.655185 24141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:50.655282 24141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:50.655479 24141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:50.665221 24141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab
--principal=oryx/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:50.666436 24141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:50.667706 24141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:50.679916 24149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:50.679857 24147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:50.680225 24146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:50.680756 24141 server_base.cc:1034] running on GCE node
I20260925 14:52:50.681327 24141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:50.682215 24141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:50.683423 24141 hybrid_clock.cc:648] HybridClock initialized: now 1790347970683389 us; error 56 us; skew 500 ppm
I20260925 14:52:50.683706 24141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:50 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347970, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:50.686662 24141 init.cc:377] Logged in from keytab as oryx/127.17.133.193@KRBTEST.COM (short username oryx)
I20260925 14:52:50.687933 24141 webserver.cc:460] Webserver started at http://127.17.133.193:41677/ using document root <none> and password file <none>
I20260925 14:52:50.688508 24141 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:50.688652 24141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:50.688948 24141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:50.691594 24141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "03e79e06fdab4fa6a55ee62e07e08231"
format_stamp: "Formatted at 2026-09-25 14:52:50 on dist-test-slave-v5vl"
server_key: "d21d7926eb67d0189d47b56853dc4c37"
server_key_iv: "2c7fdb682ef8f15d6f5ce252683221dc"
server_key_version: "encryptionkey@0"
I20260925 14:52:50.692219 24141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "03e79e06fdab4fa6a55ee62e07e08231"
format_stamp: "Formatted at 2026-09-25 14:52:50 on dist-test-slave-v5vl"
server_key: "d21d7926eb67d0189d47b56853dc4c37"
server_key_iv: "2c7fdb682ef8f15d6f5ce252683221dc"
server_key_version: "encryptionkey@0"
I20260925 14:52:50.696748 24141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 14:52:50.699812 24156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:50.701081 24141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:52:50.701231 24141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "03e79e06fdab4fa6a55ee62e07e08231"
format_stamp: "Formatted at 2026-09-25 14:52:50 on dist-test-slave-v5vl"
server_key: "d21d7926eb67d0189d47b56853dc4c37"
server_key_iv: "2c7fdb682ef8f15d6f5ce252683221dc"
server_key_version: "encryptionkey@0"
I20260925 14:52:50.701385 24141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:50.726007 24141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:50.729825 24141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:50.730118 24141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:50.731040 24141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:50.732298 24141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:50.732365 24141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:50.732515 24141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:50.732573 24141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:50.764297 24141 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:36853
I20260925 14:52:50.764348 24268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:36853 every 8 connection(s)
I20260925 14:52:50.765522 24141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:50.772851 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 24141
I20260925 14:52:50.773073 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:50.773826 17943 external_mini_cluster.cc:1428] Setting key f837530cc14dfa32b76d9f4279f6661d
Sep 25 14:52:50 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347970, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:52:50.785151 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:50.767903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:56267 (local address 127.17.133.254:40045)
0925 14:52:50.768319 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:50.768325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:50.769398 (+ 1073us) server_negotiation.cc:407] Connection header received
0925 14:52:50.771075 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:50.771079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:50.771148 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:50.771238 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:50.772639 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:50.773797 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:50.775554 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:50.776164 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:50.776291 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:50.779483 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:50.779492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:50.779499 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:50.779541 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:50.782361 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:50.783159 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:50.783162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:50.783164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:50.783265 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:50.783662 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:50.783665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:50.783666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:50.783879 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:50.783975 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:50.784760 (+ 785us) server_negotiation.cc:299] Negotiation successful
0925 14:52:50.784901 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":175,"threads_started":1}
I20260925 14:52:50.786525 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:50.768533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.193:56267)
0925 14:52:50.769247 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:50.769289 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:50.770846 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:50.771423 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:50.771430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:50.771901 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:50.772451 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:50.772467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:50.773965 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:50.773969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:50.775401 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:50.775414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:50.776367 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:50.776372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:50.776553 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:50.777278 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:50.777298 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:50.779285 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:50.782491 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:50.782502 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:50.782518 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:50.783062 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:50.783354 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:50.783361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:50.783367 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:50.783572 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:50.783990 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:50.783999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:50.784394 (+ 395us) client_negotiation.cc:769] Sending connection context
0925 14:52:50.784711 (+ 317us) client_negotiation.cc:240] Negotiation successful
0925 14:52:50.785221 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":254,"threads_started":1}
I20260925 14:52:50.787896 24269 heartbeater.cc:347] Connected to a master server at 127.17.133.254:40045
I20260925 14:52:50.788164 24269 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:50.788831 24269 heartbeater.cc:511] Master 127.17.133.254:40045 requested a full tablet report, sending...
I20260925 14:52:50.790870 24080 ts_manager.cc:194] Registered new tserver with Master: 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853)
I20260925 14:52:50.792237 24080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:56267
WARNING: no policy specified for oryx/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab.
Entry for principal oryx/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab.
I20260925 14:52:50.819432 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40045
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab
--principal=oryx/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:51.127513 24275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:51.127812 24275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:51.127987 24275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:51.134730 24275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:51.134873 24275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:51.135033 24275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:51.144187 24275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab
--principal=oryx/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:51.145442 24275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:51.146723 24275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:51.159255 24281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:51.159412 24280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:51.159387 24283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:51.160449 24275 server_base.cc:1034] running on GCE node
I20260925 14:52:51.161031 24275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:51.161917 24275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:51.163115 24275 hybrid_clock.cc:648] HybridClock initialized: now 1790347971163043 us; error 69 us; skew 500 ppm
I20260925 14:52:51.163396 24275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:51 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347971, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:51.166589 24275 init.cc:377] Logged in from keytab as oryx/127.17.133.194@KRBTEST.COM (short username oryx)
I20260925 14:52:51.168056 24275 webserver.cc:460] Webserver started at http://127.17.133.194:42215/ using document root <none> and password file <none>
I20260925 14:52:51.168843 24275 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:51.168983 24275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:51.169282 24275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:51.172748 24275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "05a38ecf8c754ae68ed0f8d7e2630e21"
format_stamp: "Formatted at 2026-09-25 14:52:51 on dist-test-slave-v5vl"
server_key: "204645cbf75d9c8bc0cbf9fc440d7988"
server_key_iv: "7757063a6e8f91758be651ded197f974"
server_key_version: "encryptionkey@0"
I20260925 14:52:51.173652 24275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "05a38ecf8c754ae68ed0f8d7e2630e21"
format_stamp: "Formatted at 2026-09-25 14:52:51 on dist-test-slave-v5vl"
server_key: "204645cbf75d9c8bc0cbf9fc440d7988"
server_key_iv: "7757063a6e8f91758be651ded197f974"
server_key_version: "encryptionkey@0"
I20260925 14:52:51.178573 24275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:52:51.181643 24290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:51.182829 24275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:52:51.182983 24275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "05a38ecf8c754ae68ed0f8d7e2630e21"
format_stamp: "Formatted at 2026-09-25 14:52:51 on dist-test-slave-v5vl"
server_key: "204645cbf75d9c8bc0cbf9fc440d7988"
server_key_iv: "7757063a6e8f91758be651ded197f974"
server_key_version: "encryptionkey@0"
I20260925 14:52:51.183154 24275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:51.195740 24275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:51.199446 24275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:51.199699 24275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:51.200555 24275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:51.201855 24275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:51.201925 24275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:51.202076 24275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:51.202127 24275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:51.234138 24275 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:41273
I20260925 14:52:51.234175 24402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:41273 every 8 connection(s)
I20260925 14:52:51.235339 24275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:51.237975 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 24275
I20260925 14:52:51.238169 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:51.238682 17943 external_mini_cluster.cc:1428] Setting key 0a6c6fe1dd77b6a1eae1d3d66e2753a2
Sep 25 14:52:51 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347971, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:52:51.254374 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:51.237737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:51299 (local address 127.17.133.254:40045)
0925 14:52:51.237891 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:51.237895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:51.239353 (+ 1458us) server_negotiation.cc:407] Connection header received
0925 14:52:51.241161 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:51.241166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:51.241248 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:51.241346 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:51.242800 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:51.243525 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:51.245112 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:51.245529 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:51.245652 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:51.249094 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:51.249098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:51.249101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:51.249129 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:51.251510 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:51.252167 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:51.252172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:51.252175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:51.252288 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:51.252741 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:51.252746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:51.252749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:51.253003 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:51.253171 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:51.253794 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 14:52:51.254004 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:52:51.255363 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:51.238403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.194:51299)
0925 14:52:51.239171 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:51.239226 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:51.240926 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:51.241562 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:51.241569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:51.242085 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:51.242644 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:51.242661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:51.243669 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:51.243674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:51.244980 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:51.244991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:51.245742 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:51.245749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:51.245965 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:51.246593 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:51.246614 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:51.248896 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:51.251639 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:51.251647 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:51.251662 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:51.252049 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:51.252407 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:51.252411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:51.252415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:51.252620 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:51.253144 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:51.253152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:51.253482 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 14:52:51.253712 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 14:52:51.254246 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":334,"threads_started":1}
I20260925 14:52:51.256867 24403 heartbeater.cc:347] Connected to a master server at 127.17.133.254:40045
WARNING: no policy specified for oryx/127.17.133.195@KRBTEST.COM; defaulting to no policy
I20260925 14:52:51.257169 24403 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:51.257921 24403 heartbeater.cc:511] Master 127.17.133.254:40045 requested a full tablet report, sending...
I20260925 14:52:51.259370 24080 ts_manager.cc:194] Registered new tserver with Master: 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.195@KRBTEST.COM" created.
I20260925 14:52:51.260205 24080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:51299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab.
Entry for principal oryx/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab.
I20260925 14:52:51.284040 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40045
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab
--principal=oryx/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:51.593032 24408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:51.593286 24408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:51.593436 24408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:51.600217 24408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:51.600334 24408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:51.600507 24408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:51.609772 24408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab
--principal=oryx/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:40045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:51.611059 24408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:51.612253 24408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:51.624411 24416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:51.624368 24413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:51.624449 24414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:51.629856 24408 server_base.cc:1034] running on GCE node
I20260925 14:52:51.630431 24408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:51.631276 24408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:51.632490 24408 hybrid_clock.cc:648] HybridClock initialized: now 1790347971632443 us; error 55 us; skew 500 ppm
I20260925 14:52:51.632794 24408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:51 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347971, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:51.635725 24408 init.cc:377] Logged in from keytab as oryx/127.17.133.195@KRBTEST.COM (short username oryx)
I20260925 14:52:51.637295 24408 webserver.cc:460] Webserver started at http://127.17.133.195:36035/ using document root <none> and password file <none>
I20260925 14:52:51.637988 24408 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:51.638111 24408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:51.638401 24408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:51.641309 24408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "dc52110daacc4bd0bcc641e389ccd486"
format_stamp: "Formatted at 2026-09-25 14:52:51 on dist-test-slave-v5vl"
server_key: "31415e5a36609b8cebc2a4ec89654f3c"
server_key_iv: "9a6c1d486f29aaf890deb47d28d06b26"
server_key_version: "encryptionkey@0"
I20260925 14:52:51.642086 24408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "dc52110daacc4bd0bcc641e389ccd486"
format_stamp: "Formatted at 2026-09-25 14:52:51 on dist-test-slave-v5vl"
server_key: "31415e5a36609b8cebc2a4ec89654f3c"
server_key_iv: "9a6c1d486f29aaf890deb47d28d06b26"
server_key_version: "encryptionkey@0"
I20260925 14:52:51.647028 24408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:52:51.650233 24423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:51.651480 24408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:51.651638 24408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "dc52110daacc4bd0bcc641e389ccd486"
format_stamp: "Formatted at 2026-09-25 14:52:51 on dist-test-slave-v5vl"
server_key: "31415e5a36609b8cebc2a4ec89654f3c"
server_key_iv: "9a6c1d486f29aaf890deb47d28d06b26"
server_key_version: "encryptionkey@0"
I20260925 14:52:51.651825 24408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:51.670135 24408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:51.674031 24408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:51.674293 24408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:51.675248 24408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:51.676680 24408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:51.676752 24408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:51.676904 24408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:51.676959 24408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:51.709281 24408 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:40247
I20260925 14:52:51.709319 24535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:40247 every 8 connection(s)
I20260925 14:52:51.710558 24408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:51.711980 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 24408
I20260925 14:52:51.712180 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:51.712735 17943 external_mini_cluster.cc:1428] Setting key 1b6b74701c4ab1a6c1e88ec6a34f6516
Sep 25 14:52:51 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347971, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:52:51.729820 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:51.713182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:50709 (local address 127.17.133.254:40045)
0925 14:52:51.713348 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:51.713351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:51.714790 (+ 1439us) server_negotiation.cc:407] Connection header received
0925 14:52:51.716748 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:51.716754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:51.716840 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:51.716937 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:51.718317 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:51.719027 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:51.720521 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:51.720978 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:51.721116 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:51.724419 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:51.724424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:51.724428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:51.724457 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:51.727282 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:51.728012 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:51.728015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:51.728017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:51.728127 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:51.728464 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:51.728466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:51.728468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:51.728760 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:51.728891 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:51.729443 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 14:52:51.729580 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:52:51.730733 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:51.713795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.195:50709)
0925 14:52:51.714652 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:51.714705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:51.716480 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:51.717160 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:51.717168 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:51.717645 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:51.718169 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:51.718184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:51.719174 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:51.719179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:51.720387 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:51.720397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:51.721178 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:51.721187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:51.721425 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:51.722006 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:51.722030 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:51.724217 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:51.727408 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:51.727418 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:51.727434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:51.727914 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:51.728222 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:51.728225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:51.728227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:51.728384 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:51.728886 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:51.728891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:51.729174 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 14:52:51.729388 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 14:52:51.729759 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":326,"threads_started":1}
I20260925 14:52:51.731839 24536 heartbeater.cc:347] Connected to a master server at 127.17.133.254:40045
I20260925 14:52:51.732105 24536 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:51.732833 24536 heartbeater.cc:511] Master 127.17.133.254:40045 requested a full tablet report, sending...
I20260925 14:52:51.734119 24080 ts_manager.cc:194] Registered new tserver with Master: dc52110daacc4bd0bcc641e389ccd486 (127.17.133.195:40247)
I20260925 14:52:51.734859 24080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:50709
I20260925 14:52:51.739449 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.133.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.253.keytab.
Entry for principal oryx/127.17.133.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.253.keytab.
I20260925 14:52:51.783991 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.253:39459
--webserver_interface=127.17.133.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.253.keytab
--principal=oryx/127.17.133.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:40045,127.17.133.253:39459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:51.795089 24269 heartbeater.cc:503] Master 127.17.133.254:40045 was elected leader, sending a full tablet report...
W20260925 14:52:52.092418 24541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:52.092818 24541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:52.092976 24541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:52.099866 24541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:52.099972 24541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:52.100065 24541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:52.100153 24541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:52.100222 24541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:52.109556 24541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:40045,127.17.133.253:39459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.253.keytab
--principal=oryx/127.17.133.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.253:39459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.133.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
W20260925 14:52:52.109871 24541 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.17.133.254:40045,127.17.133.253:39459'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 14:52:52.110827 24541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:52.112072 24541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:52.123104 24546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:52.123157 24547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:52.123631 24549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:52.124644 24541 server_base.cc:1034] running on GCE node
I20260925 14:52:52.125403 24541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:52.126844 24541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:52.128132 24541 hybrid_clock.cc:648] HybridClock initialized: now 1790347972128070 us; error 36 us; skew 500 ppm
I20260925 14:52:52.128470 24541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:52.131484 24541 init.cc:377] Logged in from keytab as oryx/127.17.133.253@KRBTEST.COM (short username oryx)
I20260925 14:52:52.132782 24541 webserver.cc:460] Webserver started at http://127.17.133.253:32823/ using document root <none> and password file <none>
I20260925 14:52:52.133409 24541 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:52.133522 24541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:52.133807 24541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:52.136691 24541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data/instance:
uuid: "703f693915d3422f99a91c1497d7b24c"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "c6a0f61459ddcc2e81e673434e7a764e"
server_key_iv: "741bbbea36a5eca07fa4b276627dd138"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.137465 24541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal/instance:
uuid: "703f693915d3422f99a91c1497d7b24c"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "c6a0f61459ddcc2e81e673434e7a764e"
server_key_iv: "741bbbea36a5eca07fa4b276627dd138"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.142320 24541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 14:52:52.145385 24556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:52.146627 24541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 14:52:52.146795 24541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal
uuid: "703f693915d3422f99a91c1497d7b24c"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "c6a0f61459ddcc2e81e673434e7a764e"
server_key_iv: "741bbbea36a5eca07fa4b276627dd138"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.146991 24541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:52.169231 24541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:52.174299 24541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:52.174538 24541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:52.192019 24541 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.253:39459
I20260925 14:52:52.192020 24607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.253:39459 every 8 connection(s)
I20260925 14:52:52.193365 24541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
I20260925 14:52:52.196888 24608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:52.201588 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 24541
I20260925 14:52:52.203173 24608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.253@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.253@KRBTEST.COM
I20260925 14:52:52.220695 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.205669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.17.133.254:40045)
0925 14:52:52.205847 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.205851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.206709 (+ 858us) server_negotiation.cc:407] Connection header received
0925 14:52:52.207146 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.207154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.207252 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.207370 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.208532 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.209244 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.210574 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.210966 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.211090 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.214264 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.214269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.214272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.214299 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.217595 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.218311 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.218318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.218324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.218480 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.218876 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.218879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.218881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.219174 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.219303 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.220082 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.220313 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:52:52.221764 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.205835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57084)
0925 14:52:52.206526 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.206563 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.206920 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.207420 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.207426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.208055 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.208393 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.208407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.209375 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.209384 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.210429 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.210439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.211166 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.211172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.211332 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.211990 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.212023 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.214081 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.217717 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.217724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.217734 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.218202 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.218588 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.218592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.218594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.218751 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.219307 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.219315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.219673 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.219993 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.220349 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":369,"threads_started":1}
I20260925 14:52:52.222177 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.203512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39548 (local address 127.17.133.253:39459)
0925 14:52:52.204297 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.204307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.204342 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:52:52.206700 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.206722 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.207016 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.207219 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.208633 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.209642 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.210770 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.211259 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.211383 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.215366 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.215378 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.215394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.215431 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.218721 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.219352 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.219359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.219361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.219519 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.219968 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.219974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.219978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.220378 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.220692 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.221493 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.221723 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":397,"threads_started":1}
I20260925 14:52:52.223692 24608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.253@KRBTEST.COM for oryx/127.17.133.253@KRBTEST.COM
I20260925 14:52:52.235379 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.224648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.0.0.1:39556)
0925 14:52:52.224868 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.224886 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.224999 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.225267 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.225271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.225599 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.225711 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.225718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.226659 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.226665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.227509 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.227518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.228157 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.228162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.228327 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.228825 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.228839 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.230590 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.233299 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.233305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.233309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.233660 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.233972 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.233977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.233980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.234089 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.234531 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.234534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.234656 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.234776 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.235032 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 14:52:52.235423 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.224645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.17.133.253:39459)
0925 14:52:52.224859 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.224864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.224950 (+ 86us) server_negotiation.cc:407] Connection header received
0925 14:52:52.225106 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.225111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.225169 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.225255 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.225835 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.226538 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.227633 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.228021 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.228153 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.230749 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.230755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.230760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.230797 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.233159 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.233761 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.233768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.233772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.233872 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.234185 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.234191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.234195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.234427 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.234554 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.234858 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.235158 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":87}
I20260925 14:52:52.240187 24608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: Bootstrap starting.
I20260925 14:52:52.243108 24608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:52.244071 24608 log.cc:824] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:52.246464 24608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: No bootstrap required, opened a new log
I20260925 14:52:52.253376 24608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } }
I20260925 14:52:52.253666 24608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:52.253751 24608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f693915d3422f99a91c1497d7b24c, State: Initialized, Role: FOLLOWER
I20260925 14:52:52.254285 24608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } }
I20260925 14:52:52.255016 24614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } } }
I20260925 14:52:52.255318 24614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:52:52.255923 24608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:52.263378 24403 heartbeater.cc:503] Master 127.17.133.254:40045 was elected leader, sending a full tablet report...
W20260925 14:52:52.263684 24625 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:52:52.263804 24625 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:52:52.266016 24541 master_runner.cc:428] Detected that this master 703f693915d3422f99a91c1497d7b24c is joining an existing cluster
I20260925 14:52:52.266110 24541 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.133.253:39459
I20260925 14:52:52.285023 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.270188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.17.133.254:40045)
0925 14:52:52.270511 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.270516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.270528 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:52:52.270567 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.270570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.270620 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.270726 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.272061 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.272844 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.274723 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.275259 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.275372 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.278015 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.278025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.278033 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.278077 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.281766 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.282662 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.282670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.282677 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.282830 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.283299 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.283307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.283312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.283676 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.283804 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.284401 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.284643 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260925 14:52:52.285117 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.269583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57094)
0925 14:52:52.270168 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.270195 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.270312 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.270746 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.270750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.271492 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.271941 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.271950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.272958 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.272969 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.274582 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.274593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.275590 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.275596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.275780 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.276473 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.276501 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.277811 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.281920 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.281927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.281934 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.282534 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.282980 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.282988 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.282994 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.283167 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.283901 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.283909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.284140 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.284293 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.284638 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"spinlock_wait_cycles":18304,"thread_start_us":1260,"threads_started":1}
I20260925 14:52:52.287416 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.269972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.17.133.253:39459)
0925 14:52:52.270365 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.270372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.271699 (+ 1327us) server_negotiation.cc:407] Connection header received
0925 14:52:52.271858 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.271862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.271943 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.272033 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.273297 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.274346 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.276693 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.277481 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.277593 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.280746 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.280752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.280757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.280790 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.284416 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.285213 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.285220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.285223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.285387 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.285824 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.285829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.285832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.286182 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.286370 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.286655 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.286978 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":279}
I20260925 14:52:52.287956 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.269704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.0.0.1:39570)
0925 14:52:52.271584 (+ 1880us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.271603 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.271738 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.272176 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.272182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.272803 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.273139 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.273155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.274543 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.274548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.276512 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.276526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.278198 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.278204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.278405 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.279205 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.279227 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.280557 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.284571 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.284575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.284579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.285093 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.285522 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.285526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.285530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.285674 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.286307 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.286312 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.286451 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.286574 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.287567 (+ 993us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1740,"thread_start_us":1551,"threads_started":1}
WARNING: no policy specified for oryx/127.17.133.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.252@KRBTEST.COM" created.
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347970, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.254@KRBTEST.COM for oryx/127.17.133.253@KRBTEST.COM
I20260925 14:52:52.313234 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.292011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39584 (local address 127.17.133.253:39459)
0925 14:52:52.292170 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.292178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.300123 (+ 7945us) server_negotiation.cc:407] Connection header received
0925 14:52:52.300462 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.300466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.300547 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.300678 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.301806 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.302496 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.303763 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.304353 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.304484 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.308008 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.308014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.308019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.308058 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.310693 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.311318 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.311325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.311329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.311455 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.311790 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.311796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.311799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.312048 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.312186 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.312643 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.312852 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:52:52.313269 24634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.292019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.0.0.1:39584)
0925 14:52:52.300007 (+ 7988us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.300044 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.300263 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.300728 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.300736 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.301432 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.301657 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.301665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.302617 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.302624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.303611 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.303627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.304505 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.304511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.304733 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.305380 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.305397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.307831 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.310810 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.310822 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.310830 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.311211 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.311553 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.311560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.311564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.311694 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.312165 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.312174 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.312348 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.312527 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.312858 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7775,"thread_start_us":7628,"threads_started":1}
I20260925 14:52:52.315606 24080 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.133.253:39459: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "703f693915d3422f99a91c1497d7b24c"
member_type: NON_VOTER
last_known_addr {
host: "127.17.133.253"
port: 39459
}
attrs {
promote: true
}
}
dest_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b"
cas_config_opid_index: -1
I20260925 14:52:52.316748 24080 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } }
I20260925 14:52:52.328091 24637 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 703f693915d3422f99a91c1497d7b24c (127.17.133.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } } }
I20260925 14:52:52.330153 24583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:52.333946 24583 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Refusing update from remote peer c1ddf08cf4ea40398ad8a455ebdd9f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 14:52:52.335191 24637 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.133.253:39459
I20260925 14:52:52.335426 24639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:52.335765 24638 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } } }
I20260925 14:52:52.336048 24638 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:52.336449 24541 master.cc:537] Master@127.17.133.253:39459 shutting down...
W20260925 14:52:52.339071 24067 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.252.keytab.
Entry for principal oryx/127.17.133.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.252.keytab.
W20260925 14:52:52.342896 24067 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b -> Peer 703f693915d3422f99a91c1497d7b24c (127.17.133.253:39459): Couldn't send request to peer 703f693915d3422f99a91c1497d7b24c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:52:52.344396 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.252:34371
--webserver_interface=127.17.133.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.252.keytab
--principal=oryx/127.17.133.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:40045,127.17.133.253:39459,127.17.133.252:34371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:52.348899 24541 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:52.349314 24541 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:52.349506 24541 tablet_replica.cc:330] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: stopping tablet replica
I20260925 14:52:52.366132 24541 master.cc:552] Master@127.17.133.253:39459 shutdown complete.
I20260925 14:52:52.366242 24541 master_runner.cc:305] Clearing existing system tablet
I20260925 14:52:52.366729 24541 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:52.366834 24541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:52.370013 24541 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 14:52:52.372916 24644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:52.373235 24541 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:52.373502 24541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:52:52.373581 24541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal
uuid: "703f693915d3422f99a91c1497d7b24c"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "c6a0f61459ddcc2e81e673434e7a764e"
server_key_iv: "741bbbea36a5eca07fa4b276627dd138"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.378567 24541 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:52.384994 24541 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:52:52.385141 24541 log.cc:1196] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 14:52:52.385555 24541 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: Deleting consensus metadata
I20260925 14:52:52.385798 24541 master_runner.cc:315] Copying system tablet from 127.17.133.254:40045
I20260925 14:52:52.388243 24541 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.133.254:40045
I20260925 14:52:52.405988 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.389167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.17.133.254:40045)
0925 14:52:52.389351 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.389357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.391495 (+ 2138us) server_negotiation.cc:407] Connection header received
0925 14:52:52.391692 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.391699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.391795 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.391898 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.392882 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.393966 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.395800 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.396440 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.396565 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.398929 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.398935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.398939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.398976 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.402480 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.403375 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.403381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.403385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.403528 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.404020 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.404025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.404029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.404437 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.404630 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.405284 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.405565 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:52:52.406203 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.389042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57096)
0925 14:52:52.391368 (+ 2326us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.391393 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.391539 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.391939 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.391944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.392429 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.392723 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.392736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.394119 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.394126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.395649 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.395662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.396622 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.396630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.396848 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.397547 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.397570 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.398703 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.402637 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.402645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.402651 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.403250 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.403652 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.403658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.403665 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.403879 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.404619 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.404628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.404926 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.405145 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.405600 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2201,"thread_start_us":2061,"threads_started":1}
I20260925 14:52:52.407434 24100 tablet_copy_service.cc:140] P c1ddf08cf4ea40398ad8a455ebdd9f6b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 703f693915d3422f99a91c1497d7b24c ({username='oryx', principal='oryx/127.17.133.253@KRBTEST.COM'} at 127.0.0.1:57096)
I20260925 14:52:52.407764 24100 tablet_copy_service.cc:161] P c1ddf08cf4ea40398ad8a455ebdd9f6b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 703f693915d3422f99a91c1497d7b24c at {username='oryx', principal='oryx/127.17.133.253@KRBTEST.COM'} at 127.0.0.1:57096: session id = 703f693915d3422f99a91c1497d7b24c-00000000000000000000000000000000
I20260925 14:52:52.412359 24100 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:52:52.414580 24541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:52:52.421242 24541 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: tablet copy: Starting download of 0 data blocks...
I20260925 14:52:52.421586 24541 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: tablet copy: Starting download of 1 WAL segments...
I20260925 14:52:52.424240 24541 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:52:52.428637 24100 tablet_copy_service.cc:342] P c1ddf08cf4ea40398ad8a455ebdd9f6b: Request end of tablet copy session 703f693915d3422f99a91c1497d7b24c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.133.253@KRBTEST.COM'} at 127.0.0.1:57096
I20260925 14:52:52.428861 24100 tablet_copy_service.cc:433] P c1ddf08cf4ea40398ad8a455ebdd9f6b: ending tablet copy session 703f693915d3422f99a91c1497d7b24c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 703f693915d3422f99a91c1497d7b24c
I20260925 14:52:52.439198 24541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:52.441638 24651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:52.446003 24652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:52.456480 24541 server_base.cc:1034] running on GCE node
W20260925 14:52:52.456864 24654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:52.457317 24541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:52.457919 24541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:52.459122 24541 hybrid_clock.cc:648] HybridClock initialized: now 1790347972459052 us; error 50 us; skew 500 ppm
I20260925 14:52:52.459324 24541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:52.462428 24541 init.cc:377] Logged in from keytab as oryx/127.17.133.253@KRBTEST.COM (short username oryx)
I20260925 14:52:52.471706 24541 webserver.cc:460] Webserver started at http://127.17.133.253:40185/ using document root <none> and password file <none>
I20260925 14:52:52.472043 24541 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:52.472138 24541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:52.474998 24541 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 14:52:52.476482 24661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:52.476892 24541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:52:52.477068 24541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal
uuid: "703f693915d3422f99a91c1497d7b24c"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "c6a0f61459ddcc2e81e673434e7a764e"
server_key_iv: "741bbbea36a5eca07fa4b276627dd138"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.477216 24541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:52.500536 24541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:52.503880 24541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:52.539379 24541 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.253:39459
I20260925 14:52:52.539402 24712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.253:39459 every 8 connection(s)
I20260925 14:52:52.540652 24541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
I20260925 14:52:52.547973 24713 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:52:52.553364 24713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: Bootstrap starting.
I20260925 14:52:52.566452 24713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:52:52.566846 24713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: Bootstrap complete.
I20260925 14:52:52.567793 24713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } }
I20260925 14:52:52.568013 24713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 703f693915d3422f99a91c1497d7b24c, State: Initialized, Role: LEARNER
I20260925 14:52:52.568233 24713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } }
I20260925 14:52:52.568962 24716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } } }
I20260925 14:52:52.569257 24713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:52.569307 24716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: This master's current role is: LEARNER
I20260925 14:52:52.578127 24727 catalog_manager.cc:1259] Loaded cluster ID: f6ee1cb7e85e4426a3353ff2e9df5f63
I20260925 14:52:52.578274 24727 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: loading cluster ID for follower catalog manager: success
I20260925 14:52:52.580575 24727 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: acquiring CA information for follower catalog manager: success
I20260925 14:52:52.582451 24727 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:52:52.737172 24536 heartbeater.cc:503] Master 127.17.133.254:40045 was elected leader, sending a full tablet report...
W20260925 14:52:52.767671 24642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:52.767935 24642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:52.768061 24642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:52.775643 24642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:52.775759 24642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:52.775803 24642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:52.775857 24642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:52.775897 24642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:52.786303 24642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:40045,127.17.133.253:39459,127.17.133.252:34371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.252.keytab
--principal=oryx/127.17.133.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.252:34371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.133.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:52.788079 24642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:52.789351 24642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:52.801431 24733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:52.801508 24735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:52.801491 24732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:52.802970 24642 server_base.cc:1034] running on GCE node
I20260925 14:52:52.803716 24642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:52.805258 24642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:52.806438 24642 hybrid_clock.cc:648] HybridClock initialized: now 1790347972806375 us; error 52 us; skew 500 ppm
I20260925 14:52:52.806777 24642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:52.809741 24642 init.cc:377] Logged in from keytab as oryx/127.17.133.252@KRBTEST.COM (short username oryx)
I20260925 14:52:52.810946 24642 webserver.cc:460] Webserver started at http://127.17.133.252:40205/ using document root <none> and password file <none>
I20260925 14:52:52.811563 24642 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:52.811712 24642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:52.812024 24642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:52.814952 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data/instance:
uuid: "9726fc4e8392492eb50bbc22256bae0e"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "f7fe63ed731512265ff9d8be200182b2"
server_key_iv: "583ba2bc110ce813627f656e62c7abc5"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.815732 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal/instance:
uuid: "9726fc4e8392492eb50bbc22256bae0e"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "f7fe63ed731512265ff9d8be200182b2"
server_key_iv: "583ba2bc110ce813627f656e62c7abc5"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.820652 24642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 14:52:52.823668 24742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:52.824985 24642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:52:52.825142 24642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal
uuid: "9726fc4e8392492eb50bbc22256bae0e"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "f7fe63ed731512265ff9d8be200182b2"
server_key_iv: "583ba2bc110ce813627f656e62c7abc5"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.825299 24642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:52.839438 24642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:52.842959 24642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:52.843202 24642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:52.860569 24642 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.252:34371
I20260925 14:52:52.860631 24793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.252:34371 every 8 connection(s)
I20260925 14:52:52.861929 24642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
I20260925 14:52:52.865471 24794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:52.869164 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 24642
I20260925 14:52:52.873005 24794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.252@KRBTEST.COM
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.252@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:52:52.891418 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.876174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.17.133.254:40045)
0925 14:52:52.876775 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.876783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.877485 (+ 702us) server_negotiation.cc:407] Connection header received
0925 14:52:52.877706 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.877713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.877789 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.877887 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.879106 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.879950 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.881659 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.882027 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.882117 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.885771 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.885775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.885778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.885806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.888837 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.889613 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.889617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.889619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.889726 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.890071 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.890074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.890076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.890341 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.890475 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.891043 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.891198 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260925 14:52:52.892454 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.876130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57106)
0925 14:52:52.877367 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.877414 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.877546 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.877971 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.877976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.878662 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.878953 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.878965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.880094 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.880097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.881529 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.881543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.882229 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.882238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.882443 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.883012 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.883050 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.885624 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.888943 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.888950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.888957 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.889484 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.889831 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.889835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.889838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.889979 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.890453 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.890458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.890661 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.890828 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.891192 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":108,"thread_start_us":821,"threads_started":1}
I20260925 14:52:52.892427 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.871363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.17.133.252:34371)
0925 14:52:52.872099 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.872110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.872147 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:52:52.873808 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.873828 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.874200 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.874569 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.876036 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.878294 (+ 2258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.879735 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.880513 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.880703 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.885343 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.885354 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.885378 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.885428 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.888308 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.889060 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.889069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.889073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.889255 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.889564 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.889569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.889574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.890141 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.890330 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.890691 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.891081 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":317,"threads_started":1}
I20260925 14:52:52.893908 24794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.252@KRBTEST.COM for oryx/127.17.133.253@KRBTEST.COM
I20260925 14:52:52.905761 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.894680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.0.0.1:39586)
0925 14:52:52.894847 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.894868 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.895250 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.895556 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.895559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.895826 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.895979 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.895990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.897385 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.897389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.898260 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.898270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.898844 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.898848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.898967 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.899490 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.899506 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.901352 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.903843 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.903847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.903849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.904169 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.904461 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.904464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.904466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.904576 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.905117 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.905120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.905226 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.905331 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.905525 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 14:52:52.905820 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.894680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.17.133.253:39459)
0925 14:52:52.895213 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.895217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.895233 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:52:52.895380 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.895384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.895445 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.895528 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.896108 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.897256 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.898384 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.898736 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.898818 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.901508 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.901513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.901516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.901544 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.903741 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.904257 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.904260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.904263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.904376 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.904717 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.904721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.904722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.904994 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.905168 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.905403 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.905542 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":297,"threads_started":1}
I20260925 14:52:52.907305 24794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.252@KRBTEST.COM for oryx/127.17.133.252@KRBTEST.COM
I20260925 14:52:52.917887 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.907913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.0.0.1:39840)
0925 14:52:52.908058 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.908078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.908158 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.908389 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.908392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.908762 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.908889 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.908897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.909731 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.909734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.910540 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.910549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.911121 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.911125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.911273 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.911747 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.911763 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.913541 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.915982 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.915985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.915988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.916311 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.916624 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.916629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.916634 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.916777 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.917212 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.917215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.917315 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.917418 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.917590 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:52:52.917932 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.907933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.17.133.252:34371)
0925 14:52:52.908080 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.908084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.908130 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:52:52.908235 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.908238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.908287 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.908361 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.908995 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.909633 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.910645 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.911011 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.911099 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.913704 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.913709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.913712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.913737 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.915865 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.916404 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.916407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.916409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.916503 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.916863 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.916866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.916868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.917118 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.917225 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.917503 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.917689 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":47}
I20260925 14:52:52.923674 24794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Bootstrap starting.
I20260925 14:52:52.927676 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.921599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.17.133.253:39459)
0925 14:52:52.921765 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.921769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.921884 (+ 115us) server_negotiation.cc:407] Connection header received
0925 14:52:52.922052 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.922057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.922153 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.922238 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:52.922680 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.923404 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.926133 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.926855 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.926976 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.927103 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.927409 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.927499 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:52:52.927855 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.921326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.0.0.1:39594)
0925 14:52:52.921792 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.921810 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.921943 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.922295 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.922299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.922324 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:52.922524 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.922536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.923528 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.923534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.925991 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.926010 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.927032 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.927038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.927244 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.927296 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.927393 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.927523 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":233,"threads_started":1}
I20260925 14:52:52.927865 24794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:52.928973 24688 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 14:52:52.929217 24794 log.cc:824] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:52.929574 24716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ddf08cf4ea40398ad8a455ebdd9f6b. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } } }
I20260925 14:52:52.929821 24716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: This master's current role is: LEARNER
I20260925 14:52:52.931649 24794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: No bootstrap required, opened a new log
I20260925 14:52:52.938190 24794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } }
I20260925 14:52:52.938910 24794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:52.938985 24794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9726fc4e8392492eb50bbc22256bae0e, State: Initialized, Role: FOLLOWER
I20260925 14:52:52.939438 24794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } }
I20260925 14:52:52.940145 24803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } } }
I20260925 14:52:52.940461 24803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:52:52.941098 24794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:52.947069 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 24141
W20260925 14:52:52.949882 24814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:52:52.949988 24814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:52:52.953588 24642 master_runner.cc:428] Detected that this master 9726fc4e8392492eb50bbc22256bae0e is joining an existing cluster
I20260925 14:52:52.953697 24642 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.133.252:34371
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.193@KRBTEST.COM
I20260925 14:52:52.975826 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.948512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.17.133.193:36853)
0925 14:52:52.950467 (+ 1955us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.950475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.950500 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:52:52.950560 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.950568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.950790 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.950937 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.952563 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.954026 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.956090 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.956910 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.957030 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.967504 (+ 10474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.967524 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.967541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.967592 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.971552 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.972420 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.972435 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.972451 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.972641 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.973154 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.973160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.973164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.973597 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.973871 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.975101 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.975365 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1786,"thread_start_us":1650,"threads_started":1}
I20260925 14:52:52.977649 24264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:52.981133 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.958081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.17.133.254:40045)
0925 14:52:52.958240 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.958247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.958770 (+ 523us) server_negotiation.cc:407] Connection header received
0925 14:52:52.958996 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.959002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.959089 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.959143 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.960736 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.962469 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.970869 (+ 8400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.971571 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.971663 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.974104 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.974111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.974115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.974153 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.977468 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.978549 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.978556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.978560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.978706 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.979337 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.979345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.979348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.979730 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.979882 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.980500 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.980746 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:52:52.982563 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.958173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57108)
0925 14:52:52.958676 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.958691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.958826 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.959343 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.959350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.960177 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.960543 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.960553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.967305 (+ 6752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.967313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.970680 (+ 3367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.970698 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.971983 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.971989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.972189 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.972815 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.972838 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.973882 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.977872 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.977879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.977885 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.978378 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.978929 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.978935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.978939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.979200 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.979960 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.979968 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.980199 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.981646 (+ 1447us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.982035 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":1,"thread_start_us":281,"threads_started":1}
I20260925 14:52:52.984555 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.958952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.17.133.253:39459)
0925 14:52:52.959108 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.959113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.959956 (+ 843us) server_negotiation.cc:407] Connection header received
0925 14:52:52.960176 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.960181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.960255 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.960326 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.962110 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.963153 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.965533 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.966196 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.966294 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.975542 (+ 9248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.975549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.975554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.975605 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.979306 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.982023 (+ 2717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.982029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.982033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.982187 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.982698 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.982704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.982707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.983021 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.983174 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.983731 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.983948 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:52:52.987499 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.959094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.0.0.1:39608)
0925 14:52:52.959848 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.959874 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.960046 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.960543 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.960547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.961626 (+ 1079us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.961947 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.961958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.963359 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.963366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.965347 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.965363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.966691 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.966698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.966918 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.969890 (+ 2972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.969921 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.973656 (+ 3735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.979495 (+ 5839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.979503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.979509 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.980047 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.982363 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.982367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.982370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.982571 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.983256 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.983260 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.983450 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.984796 (+ 1346us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.985121 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":389,"threads_started":1}
I20260925 14:52:52.990792 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.959934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.17.133.252:34371)
0925 14:52:52.960108 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.960113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.966851 (+ 6738us) server_negotiation.cc:407] Connection header received
0925 14:52:52.967164 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.967169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.967260 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.967321 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.968702 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.969874 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.973814 (+ 3940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.974522 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.974619 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.983645 (+ 9026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.983650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.983655 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.983686 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.987170 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.988699 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.988705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.988711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.988864 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.989296 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.989300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.989303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.989640 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.989778 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.990241 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.990442 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":54}
I20260925 14:52:52.991783 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.960204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.0.0.1:39854)
0925 14:52:52.966757 (+ 6553us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.966787 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.967033 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.967493 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.967498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.968151 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.968514 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.968525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.970557 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.970562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.973356 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.973370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.974891 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.974897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.975065 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.980435 (+ 5370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.980463 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.983490 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:52.987924 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.987928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.987931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.988375 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.988992 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:52.988997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:52.989001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:52.989179 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:52.989838 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:52.989842 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:52.989991 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 14:52:52.990976 (+ 985us) client_negotiation.cc:240] Negotiation successful
0925 14:52:52.991271 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6384,"thread_start_us":6658,"threads_started":1}
Sep 25 14:52:52 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347970, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.254@KRBTEST.COM for oryx/127.17.133.252@KRBTEST.COM
I20260925 14:52:53.006480 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.988224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.0.0.1:39856)
0925 14:52:52.988531 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:52.988551 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:52.988677 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:52.989907 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:52.989913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:52.990393 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:52.990604 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.990618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.993510 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.993516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.994895 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:52.994911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.996309 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.996315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:52.996501 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.997166 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:52.997188 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:52.999819 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.003795 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.003800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.003805 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.004291 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.004689 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.004694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.004697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.004851 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.005406 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.005412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.005584 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.005801 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.006083 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260925 14:52:53.007359 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.988860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.17.133.252:34371)
0925 14:52:52.989483 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.989491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.989516 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:52:52.989620 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.989629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.989714 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.990768 (+ 1054us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.991403 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.993328 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.995046 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.996146 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.996296 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.000060 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.000069 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.000075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.000112 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.003668 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.004406 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.004412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.004415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.004571 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.004958 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.004962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.004966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.005281 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.005415 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.005912 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.006111 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":288,"threads_started":1}
I20260925 14:52:53.008733 24080 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.133.252:34371: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e"
member_type: NON_VOTER
last_known_addr {
host: "127.17.133.252"
port: 34371
}
attrs {
promote: true
}
}
dest_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b"
cas_config_opid_index: 5
I20260925 14:52:53.009732 24080 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } }
I20260925 14:52:53.011369 24800 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9726fc4e8392492eb50bbc22256bae0e (127.17.133.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.013351 24688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 LEARNER]: Refusing update from remote peer c1ddf08cf4ea40398ad8a455ebdd9f6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:52:53.014487 24769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:53.019273 24800 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.133.252:34371
I20260925 14:52:53.031457 24826 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 14:52:53.034247 24688 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9726fc4e8392492eb50bbc22256bae0e (127.17.133.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.038399 24716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.038669 24716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: This master's current role is: LEARNER
I20260925 14:52:53.039855 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: NON_VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: true } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.040273 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.040927 24642 master.cc:537] Master@127.17.133.252:34371 shutting down...
I20260925 14:52:53.042210 24769 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 FOLLOWER]: Refusing update from remote peer c1ddf08cf4ea40398ad8a455ebdd9f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:52:53.043489 24800 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: attempting to promote NON_VOTER 703f693915d3422f99a91c1497d7b24c to VOTER
I20260925 14:52:53.044214 24800 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } }
I20260925 14:52:53.056416 24642 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:53.056921 24642 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:53.057174 24642 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: stopping tablet replica
I20260925 14:52:53.073194 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.072452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.0.0.1:39870)
0925 14:52:53.072842 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.073011 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260925 14:52:53.074700 24642 master.cc:552] Master@127.17.133.252:34371 shutdown complete.
I20260925 14:52:53.074776 24642 master_runner.cc:305] Clearing existing system tablet
I20260925 14:52:53.075178 24642 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.075269 24642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.078727 24688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 LEARNER]: Refusing update from remote peer c1ddf08cf4ea40398ad8a455ebdd9f6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 14:52:53.078971 24642 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:52:53.079242 24829 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 14:52:53.080054 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.079631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.0.0.1:39874)
0925 14:52:53.079791 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.079872 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:52:53.080364 24067 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b -> Peer 9726fc4e8392492eb50bbc22256bae0e (127.17.133.252:34371): Couldn't send request to peer 9726fc4e8392492eb50bbc22256bae0e. Status: Network error: Client connection negotiation failed: client connection to 127.17.133.252:34371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:52:53.082319 24827 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 703f693915d3422f99a91c1497d7b24c (127.17.133.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.086318 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.086314 24832 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.086592 24835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.086629 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.086689 24832 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.086900 24642 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.087146 24642 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.007s
I20260925 14:52:53.087252 24642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal
uuid: "9726fc4e8392492eb50bbc22256bae0e"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "f7fe63ed731512265ff9d8be200182b2"
server_key_iv: "583ba2bc110ce813627f656e62c7abc5"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.087479 24688 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 703f693915d3422f99a91c1497d7b24c (127.17.133.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.091094 24716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.091320 24716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:52:53.091928 24642 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:53.098383 24642 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:52:53.098554 24642 log.cc:1196] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 14:52:53.098973 24642 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Deleting consensus metadata
I20260925 14:52:53.099222 24642 master_runner.cc:315] Copying system tablet from 127.17.133.254:40045
I20260925 14:52:53.101609 24642 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: tablet copy: Beginning tablet copy session from remote peer at address 127.17.133.254:40045
I20260925 14:52:53.116719 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.102359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.17.133.254:40045)
0925 14:52:53.102571 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.102578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.103015 (+ 437us) server_negotiation.cc:407] Connection header received
0925 14:52:53.103191 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.103197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.103272 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.103358 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.104815 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.105930 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.107758 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.108350 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.108466 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.110575 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.110582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.110587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.110621 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.113951 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.114676 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.114681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.114685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.114823 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.115227 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.115232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.115236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.115554 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.115686 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.116148 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.116345 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 14:52:53.116933 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.102307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57122)
0925 14:52:53.102882 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.102902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.103061 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.103400 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.103405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.103932 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:53.104672 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.104686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.106085 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.106090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.107579 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.107597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.108521 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.108527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.108727 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.109361 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:53.109383 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:53.110369 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.114080 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.114086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.114090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.114561 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.114929 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.114936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.114943 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.115116 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.115670 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.115675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.115850 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.116015 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.116325 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":370,"threads_started":1}
I20260925 14:52:53.117528 24100 tablet_copy_service.cc:140] P c1ddf08cf4ea40398ad8a455ebdd9f6b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9726fc4e8392492eb50bbc22256bae0e ({username='oryx', principal='oryx/127.17.133.252@KRBTEST.COM'} at 127.0.0.1:57122)
I20260925 14:52:53.117692 24100 tablet_copy_service.cc:161] P c1ddf08cf4ea40398ad8a455ebdd9f6b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9726fc4e8392492eb50bbc22256bae0e at {username='oryx', principal='oryx/127.17.133.252@KRBTEST.COM'} at 127.0.0.1:57122: session id = 9726fc4e8392492eb50bbc22256bae0e-00000000000000000000000000000000
I20260925 14:52:53.121606 24100 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:52:53.123541 24642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:52:53.124441 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 24141
I20260925 14:52:53.130231 24642 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: tablet copy: Starting download of 0 data blocks...
I20260925 14:52:53.130477 24642 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: tablet copy: Starting download of 1 WAL segments...
I20260925 14:52:53.134097 24642 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:52:53.138850 24100 tablet_copy_service.cc:342] P c1ddf08cf4ea40398ad8a455ebdd9f6b: Request end of tablet copy session 9726fc4e8392492eb50bbc22256bae0e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.133.252@KRBTEST.COM'} at 127.0.0.1:57122
I20260925 14:52:53.138978 24100 tablet_copy_service.cc:433] P c1ddf08cf4ea40398ad8a455ebdd9f6b: ending tablet copy session 9726fc4e8392492eb50bbc22256bae0e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9726fc4e8392492eb50bbc22256bae0e
I20260925 14:52:53.147750 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:36853
--local_ip_for_outbound_sockets=127.17.133.193
--tserver_master_addrs=127.17.133.254:40045
--webserver_port=41677
--webserver_interface=127.17.133.193
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab
--principal=oryx/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.133.254:40045,127.17.133.253:39459
--tserver_master_addrs=127.17.133.254:40045,127.17.133.253:39459,127.17.133.252:34371 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:53.150265 24642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:53.152894 24842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.154222 24843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.167379 24847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.172534 24642 server_base.cc:1034] running on GCE node
I20260925 14:52:53.173026 24642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:53.183629 24642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:53.184823 24642 hybrid_clock.cc:648] HybridClock initialized: now 1790347973184780 us; error 36 us; skew 500 ppm
I20260925 14:52:53.185057 24642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:53 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:53.187201 24642 init.cc:377] Logged in from keytab as oryx/127.17.133.252@KRBTEST.COM (short username oryx)
I20260925 14:52:53.193342 24642 webserver.cc:460] Webserver started at http://127.17.133.252:44671/ using document root <none> and password file <none>
I20260925 14:52:53.193676 24642 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.193773 24642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.197073 24642 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:52:53.198848 24853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.199309 24642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:53.199466 24642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal
uuid: "9726fc4e8392492eb50bbc22256bae0e"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-v5vl"
server_key: "f7fe63ed731512265ff9d8be200182b2"
server_key_iv: "583ba2bc110ce813627f656e62c7abc5"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.199710 24642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.207906 24642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:53.211861 24642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:53.240760 24642 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.252:34371
I20260925 14:52:53.240828 24904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.252:34371 every 8 connection(s)
I20260925 14:52:53.241879 24642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
I20260925 14:52:53.247020 24905 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:52:53.250952 24905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Bootstrap starting.
I20260925 14:52:53.269212 24905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:52:53.269685 24905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Bootstrap complete.
I20260925 14:52:53.270795 24905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } }
I20260925 14:52:53.271085 24905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9726fc4e8392492eb50bbc22256bae0e, State: Initialized, Role: LEARNER
I20260925 14:52:53.271347 24905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } }
I20260925 14:52:53.272439 24905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:53.272148 24908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.272832 24908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: This master's current role is: LEARNER
I20260925 14:52:53.285713 24919 catalog_manager.cc:1259] Loaded cluster ID: f6ee1cb7e85e4426a3353ff2e9df5f63
I20260925 14:52:53.285804 24919 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: loading cluster ID for follower catalog manager: success
I20260925 14:52:53.289260 24919 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: acquiring CA information for follower catalog manager: success
I20260925 14:52:53.291345 24919 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 14:52:53.504024 24844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:53.504349 24844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:53.504514 24844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:53.511891 24844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:53.512027 24844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:53.512203 24844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:53.521771 24844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab
--principal=oryx/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:36853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=41677
--tserver_master_addrs=127.17.133.254:40045,127.17.133.253:39459,127.17.133.252:34371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:53.523203 24844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:53.524978 24844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:52:53.534837 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.527949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.0.0.1:39884)
0925 14:52:53.528134 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.528155 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.528284 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.529077 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.529082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.529107 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:53.529307 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.529317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.530609 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.530617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.532404 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.532421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.533614 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.533622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.533846 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.533884 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.533997 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.534142 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 14:52:53.534906 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.528134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.17.133.252:34371)
0925 14:52:53.528774 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.528787 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.528802 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:52:53.528860 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.528863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.528929 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.529055 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:53.529458 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.530466 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.532549 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.533440 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.533643 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.533838 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.534017 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.534142 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":374,"threads_started":1}
I20260925 14:52:53.535823 24880 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 14:52:53.536417 24908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ddf08cf4ea40398ad8a455ebdd9f6b. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.536761 24908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: This master's current role is: LEARNER
W20260925 14:52:53.538163 24925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.538189 24928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.540894 24926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.541945 24844 server_base.cc:1034] running on GCE node
I20260925 14:52:53.542510 24844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:53.543419 24844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:53.544651 24844 hybrid_clock.cc:648] HybridClock initialized: now 1790347973544589 us; error 53 us; skew 500 ppm
I20260925 14:52:53.544945 24844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:53 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:53.547816 24844 init.cc:377] Logged in from keytab as oryx/127.17.133.193@KRBTEST.COM (short username oryx)
I20260925 14:52:53.549283 24844 webserver.cc:460] Webserver started at http://127.17.133.193:41677/ using document root <none> and password file <none>
I20260925 14:52:53.549914 24844 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.550047 24844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.554908 24844 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 14:52:53.557619 24935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.558907 24844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:52:53.559062 24844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "03e79e06fdab4fa6a55ee62e07e08231"
format_stamp: "Formatted at 2026-09-25 14:52:50 on dist-test-slave-v5vl"
server_key: "d21d7926eb67d0189d47b56853dc4c37"
server_key_iv: "2c7fdb682ef8f15d6f5ce252683221dc"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.559954 24844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.585388 24844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:53.590188 24844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:53.590473 24844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:53.591444 24844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:53.592808 24844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:53.592919 24844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.593034 24844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:53.593124 24844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.626538 24844 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:36853
I20260925 14:52:53.626612 25047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:36853 every 8 connection(s)
I20260925 14:52:53.627869 24844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:53.633229 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 24844
I20260925 14:52:53.633994 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 24275
Sep 25 14:52:53 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.194@KRBTEST.COM
Sep 25 14:52:53 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
Sep 25 14:52:53 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.253@KRBTEST.COM
Sep 25 14:52:53 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.252@KRBTEST.COM
I20260925 14:52:53.668809 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.635498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.17.133.194:41273)
0925 14:52:53.644304 (+ 8806us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.644310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.644814 (+ 504us) server_negotiation.cc:407] Connection header received
0925 14:52:53.645024 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.645036 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.645313 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.645466 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.647428 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.648986 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.650856 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.651693 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.658708 (+ 7015us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.658989 (+ 281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.659002 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.659021 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.659059 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.664607 (+ 5548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.665506 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.665514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.665518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.665653 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.666111 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.666116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.666120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.666542 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.667936 (+ 1394us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.668087 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.668358 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8675,"thread_start_us":8562,"threads_started":1}
I20260925 14:52:53.669108 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.645833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:40069 (local address 127.17.133.252:34371)
0925 14:52:53.646030 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.646036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.647899 (+ 1863us) server_negotiation.cc:407] Connection header received
0925 14:52:53.648660 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.648666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.648750 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.648822 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.651854 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.652965 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.656056 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.656748 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.656846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.662649 (+ 5803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.662654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.662660 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.662704 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.665882 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.667086 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.667091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.667093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.667221 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.667667 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.667672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.667675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.667977 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.668092 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.668622 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.668830 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:52:53.669346 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.630383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:52215 (local address 127.17.133.254:40045)
0925 14:52:53.643904 (+ 13521us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.643908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.643927 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:52:53.648662 (+ 4735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.648668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.648751 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.648824 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.651856 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.653002 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.656056 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.656748 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.656846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.661989 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.661996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.662000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.662040 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.666237 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.667188 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.667194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.667197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.667331 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.667804 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.667809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.667814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.668140 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.668261 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.668845 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.669048 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13409,"thread_start_us":13322,"threads_started":1}
I20260925 14:52:53.670174 24398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:53.670224 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.631169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.17.133.193:51267)
0925 14:52:53.645466 (+ 14297us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.645493 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.648278 (+ 2785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.649262 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.649277 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.650417 (+ 1140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:53.651647 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.651674 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.653624 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.653639 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.655876 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.655891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.656992 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.656999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.657298 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.658096 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:53.658128 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:53.661832 (+ 3704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.665653 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.665664 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.665680 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.666297 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.666692 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.666696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.666700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.666958 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.667575 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.667585 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.668003 (+ 418us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.668302 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.668839 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14106,"mutex_wait_us":348,"spinlock_wait_cycles":56704,"thread_start_us":13747,"threads_started":1}
I20260925 14:52:53.670668 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.645763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.17.133.193:40069)
0925 14:52:53.647808 (+ 2045us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.647825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.648277 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.649262 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.649276 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.650490 (+ 1214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:53.651654 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.651674 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.653185 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.653204 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.655895 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.655909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.657199 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.657204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.657479 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.658398 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:53.658427 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:53.662436 (+ 4009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.666018 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.666026 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.666031 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.666958 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.667371 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.667377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.667380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.667551 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.668182 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.668187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.668313 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.669961 (+ 1648us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.670237 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1938,"mutex_wait_us":489,"spinlock_wait_cycles":20096,"thread_start_us":1786,"threads_started":1}
I20260925 14:52:53.672071 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.630443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:51267 (local address 127.17.133.253:39459)
0925 14:52:53.648079 (+ 17636us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.648085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.648113 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:52:53.648749 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.648754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.648838 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.649558 (+ 720us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.651851 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.653452 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.656045 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.656751 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.657904 (+ 1153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.662033 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.662039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.662044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.662086 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.665501 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.666416 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.666422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.666426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.666566 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.667095 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.667099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.667103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.667433 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.671053 (+ 3620us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.671168 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.671446 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17530,"thread_start_us":17422,"threads_started":1}
I20260925 14:52:53.673062 25048 heartbeater.cc:347] Connected to a master server at 127.17.133.252:34371
I20260925 14:52:53.673543 25048 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:53.674494 25048 heartbeater.cc:511] Master 127.17.133.252:34371 requested a full tablet report, sending...
I20260925 14:52:53.675532 25050 heartbeater.cc:347] Connected to a master server at 127.17.133.253:39459
I20260925 14:52:53.675662 25050 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:53.676043 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.631169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.193:52215)
0925 14:52:53.642985 (+ 11816us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.643052 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.648277 (+ 5225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.649262 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.649276 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.650414 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:53.651654 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.651674 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.653186 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.653191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.655895 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.655909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.657178 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.657185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.657476 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.658273 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:53.658296 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:53.661712 (+ 3416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.666394 (+ 4682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.666398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.666401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.667068 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.667478 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.667487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.667494 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.667666 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.668352 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.668357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.668524 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.669599 (+ 1075us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.675511 (+ 5912us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11436,"mutex_wait_us":571,"spinlock_wait_cycles":16512,"thread_start_us":11169,"threads_started":1}
I20260925 14:52:53.676105 25050 heartbeater.cc:511] Master 127.17.133.253:39459 requested a full tablet report, sending...
I20260925 14:52:53.676919 25049 heartbeater.cc:347] Connected to a master server at 127.17.133.254:40045
I20260925 14:52:53.677028 25049 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:53.677037 24870 ts_manager.cc:194] Registered new tserver with Master: 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853)
I20260925 14:52:53.677417 25049 heartbeater.cc:511] Master 127.17.133.254:40045 requested a full tablet report, sending...
I20260925 14:52:53.677805 24678 ts_manager.cc:194] Registered new tserver with Master: 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853)
I20260925 14:52:53.678608 24080 ts_manager.cc:194] Re-registered known tserver with Master: 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853)
I20260925 14:52:53.679751 24080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:52215
I20260925 14:52:53.758172 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 24275
I20260925 14:52:53.775168 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:41273
--local_ip_for_outbound_sockets=127.17.133.194
--tserver_master_addrs=127.17.133.254:40045
--webserver_port=42215
--webserver_interface=127.17.133.194
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab
--principal=oryx/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.133.254:40045,127.17.133.253:39459
--tserver_master_addrs=127.17.133.254:40045,127.17.133.253:39459,127.17.133.252:34371 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:53.928473 24831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: NON_VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: true } } }
I20260925 14:52:53.928949 24831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.930351 25061 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: attempting to promote NON_VOTER 9726fc4e8392492eb50bbc22256bae0e to VOTER
I20260925 14:52:53.930887 25061 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } }
I20260925 14:52:53.932606 24688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Refusing update from remote peer c1ddf08cf4ea40398ad8a455ebdd9f6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 14:52:53.932626 24880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 LEARNER]: Refusing update from remote peer c1ddf08cf4ea40398ad8a455ebdd9f6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 14:52:53.933164 25061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 14:52:53.933531 24831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 14:52:53.935855 25062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9726fc4e8392492eb50bbc22256bae0e (127.17.133.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:53.937168 24688 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9726fc4e8392492eb50bbc22256bae0e (127.17.133.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:53.939296 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:53.939574 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.939774 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:53.940018 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.941130 25065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:53.941483 25065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:52:53.941685 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:53.941941 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.942255 24880 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9726fc4e8392492eb50bbc22256bae0e (127.17.133.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:53.945024 24908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:53.945287 24908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:52:54.093707 25060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:54.094014 25060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:54.094197 25060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:54.101445 25060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:54.101574 25060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:54.101747 25060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:52:54.111313 25060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab
--principal=oryx/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:41273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=42215
--tserver_master_addrs=127.17.133.254:40045,127.17.133.253:39459,127.17.133.252:34371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:54.112639 25060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:54.114003 25060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:54.126750 25070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.126785 25071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.126785 25073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:54.127691 25060 server_base.cc:1034] running on GCE node
I20260925 14:52:54.128242 25060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:54.129146 25060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:54.130384 25060 hybrid_clock.cc:648] HybridClock initialized: now 1790347974130340 us; error 35 us; skew 500 ppm
I20260925 14:52:54.130713 25060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:54 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:54.135053 25060 init.cc:377] Logged in from keytab as oryx/127.17.133.194@KRBTEST.COM (short username oryx)
I20260925 14:52:54.136850 25060 webserver.cc:460] Webserver started at http://127.17.133.194:42215/ using document root <none> and password file <none>
I20260925 14:52:54.137718 25060 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:54.137880 25060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:54.143405 25060 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 14:52:54.146000 25080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.147248 25060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:52:54.147414 25060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "05a38ecf8c754ae68ed0f8d7e2630e21"
format_stamp: "Formatted at 2026-09-25 14:52:51 on dist-test-slave-v5vl"
server_key: "204645cbf75d9c8bc0cbf9fc440d7988"
server_key_iv: "7757063a6e8f91758be651ded197f974"
server_key_version: "encryptionkey@0"
I20260925 14:52:54.148334 25060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:54.171775 25060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:54.177153 25060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:54.177498 25060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:54.178717 25060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:54.179899 25060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:54.179978 25060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.180060 25060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:54.180112 25060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.210095 25060 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:41273
I20260925 14:52:54.210175 25192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:41273 every 8 connection(s)
I20260925 14:52:54.211452 25060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:54.215570 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 25060
I20260925 14:52:54.215814 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 24408
Sep 25 14:52:54 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
Sep 25 14:52:54 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.252@KRBTEST.COM
Sep 25 14:52:54 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.195@KRBTEST.COM
Sep 25 14:52:54 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.253@KRBTEST.COM
I20260925 14:52:54.247548 25197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.214182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:44151 (local address 127.17.133.254:40045)
0925 14:52:54.214664 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.214672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.215803 (+ 1131us) server_negotiation.cc:407] Connection header received
0925 14:52:54.219079 (+ 3276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.219087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.219182 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.219378 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.222168 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.224016 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.229493 (+ 5477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.229893 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.230060 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.235465 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.235469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.235472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.235501 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.239648 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.242413 (+ 2765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.242419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.242422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.242577 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.244723 (+ 2146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.244728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.244731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.245058 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.245193 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.246935 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.247160 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":211,"threads_started":1}
I20260925 14:52:54.249603 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.214755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.194:44151)
0925 14:52:54.215637 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.215690 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.218715 (+ 3025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.220148 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.220161 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.221071 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:54.221972 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.221999 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.226663 (+ 4664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.226672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.229340 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.229357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.230119 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.230129 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.230417 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.231259 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:54.231298 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:54.235277 (+ 3979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:54.240021 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.240029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.240044 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.240725 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.242689 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.242696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.242701 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.243721 (+ 1020us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.245627 (+ 1906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:54.245636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:54.246115 (+ 479us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.246847 (+ 732us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.248174 (+ 1327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":1807,"threads_started":1}
I20260925 14:52:54.251029 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.222806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:56109 (local address 127.17.133.252:34371)
0925 14:52:54.229229 (+ 6423us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.229233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.229248 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:52:54.229293 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.229296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.229349 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.230295 (+ 946us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.231556 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.232712 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.234577 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.235200 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.235314 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.241518 (+ 6204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.241524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.241529 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.241564 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.245810 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.248166 (+ 2356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.248170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.248174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.248301 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.248692 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.248696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.248700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.248999 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.249143 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.250456 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.250662 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6324,"thread_start_us":6243,"threads_started":1}
I20260925 14:52:54.251091 25194 heartbeater.cc:347] Connected to a master server at 127.17.133.254:40045
I20260925 14:52:54.251344 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.224787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.17.133.194:56109)
0925 14:52:54.228569 (+ 3782us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.228605 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.228710 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.230450 (+ 1740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.230454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.231026 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:54.231365 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.231378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.232889 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.232896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.234428 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.234441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.236300 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.236304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.236513 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.237523 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:54.237546 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:54.239864 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:54.245955 (+ 6091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.245958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.245961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.248014 (+ 2053us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.248418 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.248422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.248425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.248551 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.249210 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:54.249218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:54.249354 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.249481 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.249758 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3671,"thread_start_us":3490,"threads_started":1}
I20260925 14:52:54.251398 25194 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:54.252007 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.217010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.17.133.195:40247)
0925 14:52:54.223336 (+ 6326us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.223347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.223382 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:52:54.223473 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.223484 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.223779 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.229647 (+ 5868us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.229972 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.231922 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.234458 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.235540 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.235687 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.241875 (+ 6188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.241887 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.241900 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.241940 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.247119 (+ 5179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.248370 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.248378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.248382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.248537 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.248974 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.248979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.248983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.249328 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.249580 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.250965 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.251618 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6153,"thread_start_us":6024,"threads_started":1}
I20260925 14:52:54.252357 25194 heartbeater.cc:511] Master 127.17.133.254:40045 requested a full tablet report, sending...
I20260925 14:52:54.252431 25193 heartbeater.cc:347] Connected to a master server at 127.17.133.252:34371
I20260925 14:52:54.252565 25193 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:54.253005 25193 heartbeater.cc:511] Master 127.17.133.252:34371 requested a full tablet report, sending...
I20260925 14:52:54.253974 24531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:54.253978 24080 ts_manager.cc:194] Re-registered known tserver with Master: 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273)
I20260925 14:52:54.254123 24870 ts_manager.cc:194] Registered new tserver with Master: 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273)
I20260925 14:52:54.255116 24080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:44151
I20260925 14:52:54.259306 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.216402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.17.133.194:49421)
0925 14:52:54.222588 (+ 6186us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.222608 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.222719 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.236566 (+ 13847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.236572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.237615 (+ 1043us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:54.237971 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.237983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.240014 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.240019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.241308 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.241318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.242243 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.242251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.242445 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.242997 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:54.243019 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:54.245479 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:54.256039 (+ 10560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.256047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.256054 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.256739 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.257297 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.257305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.257311 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.257507 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.258344 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:54.258350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:54.258496 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.258636 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.258934 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6038,"thread_start_us":5860,"threads_started":1}
I20260925 14:52:54.260085 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.214265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:49421 (local address 127.17.133.253:39459)
0925 14:52:54.236222 (+ 21957us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.236229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.236247 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:52:54.236314 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.236320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.236434 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.236544 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.238150 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.239846 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.241434 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.242056 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.244774 (+ 2718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.247382 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.247388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.247393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.247435 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.255827 (+ 8392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.256959 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.256968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.256974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.257153 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.257672 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.257679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.257685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.258159 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.258392 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.258809 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.259105 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21839,"thread_start_us":21720,"threads_started":1}
I20260925 14:52:54.260944 25195 heartbeater.cc:347] Connected to a master server at 127.17.133.253:39459
I20260925 14:52:54.261083 25195 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:54.261505 25195 heartbeater.cc:511] Master 127.17.133.253:39459 requested a full tablet report, sending...
I20260925 14:52:54.262566 24678 ts_manager.cc:194] Registered new tserver with Master: 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273)
I20260925 14:52:54.341483 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 24408
I20260925 14:52:54.357831 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:40247
--local_ip_for_outbound_sockets=127.17.133.195
--tserver_master_addrs=127.17.133.254:40045
--webserver_port=36035
--webserver_interface=127.17.133.195
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab
--principal=oryx/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.133.254:40045,127.17.133.253:39459
--tserver_master_addrs=127.17.133.254:40045,127.17.133.253:39459,127.17.133.252:34371 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:54.675495 25206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:54.675791 25206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:54.675956 25206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 14:52:54.682392 25049 heartbeater.cc:503] Master 127.17.133.254:40045 was elected leader, sending a full tablet report...
W20260925 14:52:54.684072 25206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:54.684202 25206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:54.684389 25206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:52:54.693882 25206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab
--principal=oryx/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:40247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=36035
--tserver_master_addrs=127.17.133.254:40045,127.17.133.253:39459,127.17.133.252:34371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:54.695271 25206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:54.696635 25206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:54.709813 25211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.709903 25214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.709856 25212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:54.710940 25206 server_base.cc:1034] running on GCE node
I20260925 14:52:54.711498 25206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:54.712393 25206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:54.713598 25206 hybrid_clock.cc:648] HybridClock initialized: now 1790347974713528 us; error 65 us; skew 500 ppm
I20260925 14:52:54.713909 25206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:54 dist-test-slave-v5vl krb5kdc[24035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:54.717180 25206 init.cc:377] Logged in from keytab as oryx/127.17.133.195@KRBTEST.COM (short username oryx)
I20260925 14:52:54.718655 25206 webserver.cc:460] Webserver started at http://127.17.133.195:36035/ using document root <none> and password file <none>
I20260925 14:52:54.719326 25206 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:54.719477 25206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:54.725106 25206 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 14:52:54.727587 25221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.728914 25206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:54.729087 25206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "dc52110daacc4bd0bcc641e389ccd486"
format_stamp: "Formatted at 2026-09-25 14:52:51 on dist-test-slave-v5vl"
server_key: "31415e5a36609b8cebc2a4ec89654f3c"
server_key_iv: "9a6c1d486f29aaf890deb47d28d06b26"
server_key_version: "encryptionkey@0"
I20260925 14:52:54.730048 25206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:54.754251 25206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:54.758065 25206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:54.758337 25206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:54.759253 25206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:54.760569 25206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:54.760731 25206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.760846 25206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:54.760926 25206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.793001 25206 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:40247
I20260925 14:52:54.793079 25333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:40247 every 8 connection(s)
I20260925 14:52:54.794387 25206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:54.796361 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 25206
Sep 25 14:52:54 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
Sep 25 14:52:54 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.252@KRBTEST.COM
Sep 25 14:52:54 dist-test-slave-v5vl krb5kdc[24035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.253@KRBTEST.COM
I20260925 14:52:54.842407 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.798689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:55095 (local address 127.17.133.254:40045)
0925 14:52:54.821136 (+ 22447us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.821144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.821163 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:52:54.821217 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.821221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.821292 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.821390 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.823058 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.824263 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.827372 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.827984 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.828071 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.834952 (+ 6881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.834961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.834967 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.835009 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.838734 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.839571 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.839579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.839585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.839754 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.840247 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.840255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.840261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.840714 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.840854 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.841834 (+ 980us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.842051 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22310,"thread_start_us":22229,"threads_started":1}
I20260925 14:52:54.843710 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.799080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.195:55095)
0925 14:52:54.818174 (+ 19094us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.818195 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.818323 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.821906 (+ 3583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.821912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.822474 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:54.822898 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.822912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.824475 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.824479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.827240 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.827248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.828330 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.828337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.828716 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.829692 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:54.829724 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:54.833701 (+ 3977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:54.838999 (+ 5298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.839003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.839007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.839462 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.839846 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.839852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.839855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.840109 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.840934 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:54.840944 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:54.841405 (+ 461us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.841743 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.842256 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18974,"mutex_wait_us":250,"thread_start_us":21380,"threads_started":1}
I20260925 14:52:54.845264 25335 heartbeater.cc:347] Connected to a master server at 127.17.133.254:40045
I20260925 14:52:54.845671 25335 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:54.846824 25335 heartbeater.cc:511] Master 127.17.133.254:40045 requested a full tablet report, sending...
I20260925 14:52:54.847223 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.797966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.17.133.195:40519)
0925 14:52:54.799040 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.799114 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.802601 (+ 3487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.820435 (+ 17834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.820445 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.821572 (+ 1127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:54.822801 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.822831 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.824915 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.824924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.827214 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.827232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.828567 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.828576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.828943 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.829842 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:54.829872 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:54.834084 (+ 4212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:54.843548 (+ 9464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.843553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.843556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.844036 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.845073 (+ 1037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.845078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.845083 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.845273 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.845920 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:54.845924 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:54.846109 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.846276 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.846656 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":156,"spinlock_wait_cycles":16128,"thread_start_us":3075,"threads_started":1}
I20260925 14:52:54.847813 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.805186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:35013 (local address 127.17.133.252:34371)
0925 14:52:54.817572 (+ 12386us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.817582 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.820966 (+ 3384us) server_negotiation.cc:407] Connection header received
0925 14:52:54.821215 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.821220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.821294 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.821373 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.823019 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.824261 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.827220 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.827852 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.827972 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.833739 (+ 5767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.833748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.833756 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.833802 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.838031 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.840644 (+ 2613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.840650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.840653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.840794 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.845469 (+ 4675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.845474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.845477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.845797 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.845929 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.847213 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.847459 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12200,"thread_start_us":12412,"threads_started":1}
I20260925 14:52:54.848042 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.805088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.17.133.195:35013)
0925 14:52:54.820839 (+ 15751us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.820870 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.821064 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.821585 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.821591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.822300 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:54.822798 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.822829 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.824474 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.824479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.827073 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.827093 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.828274 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.828281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.828715 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.829988 (+ 1273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:54.830015 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:54.833481 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:54.838216 (+ 4735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.838227 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.838245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.838888 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.844303 (+ 5415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.844308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.844311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.844459 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.845977 (+ 1518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:54.845981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:54.846152 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.846292 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.847618 (+ 1326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15608,"mutex_wait_us":1215,"spinlock_wait_cycles":18304,"thread_start_us":12793,"threads_started":1}
I20260925 14:52:54.849447 25334 heartbeater.cc:347] Connected to a master server at 127.17.133.252:34371
I20260925 14:52:54.849538 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.798689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:40519 (local address 127.17.133.253:39459)
0925 14:52:54.819750 (+ 21061us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.819760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.819791 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:52:54.819871 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.819879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.819996 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.820153 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.823114 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.824682 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.827432 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.828319 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.829375 (+ 1056us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.836760 (+ 7385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.836771 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.836779 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.836844 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.843390 (+ 6546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.844791 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.844796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.844799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.844954 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.845444 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.845449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.845452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.845784 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.848506 (+ 2722us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.848689 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.848975 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20894,"thread_start_us":20808,"threads_started":1}
I20260925 14:52:54.849594 25334 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:54.850132 25334 heartbeater.cc:511] Master 127.17.133.252:34371 requested a full tablet report, sending...
I20260925 14:52:54.850884 25336 heartbeater.cc:347] Connected to a master server at 127.17.133.253:39459
I20260925 14:52:54.851009 25336 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:54.851541 25336 heartbeater.cc:511] Master 127.17.133.253:39459 requested a full tablet report, sending...
I20260925 14:52:54.852546 24870 ts_manager.cc:194] Registered new tserver with Master: dc52110daacc4bd0bcc641e389ccd486 (127.17.133.195:40247)
I20260925 14:52:54.852838 24678 ts_manager.cc:194] Registered new tserver with Master: dc52110daacc4bd0bcc641e389ccd486 (127.17.133.195:40247)
I20260925 14:52:54.854085 24080 ts_manager.cc:194] Re-registered known tserver with Master: dc52110daacc4bd0bcc641e389ccd486 (127.17.133.195:40247)
I20260925 14:52:54.855643 24080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:55095
I20260925 14:52:54.870931 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.832920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.17.133.254:40045)
0925 14:52:54.834137 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.834142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.853156 (+ 19014us) server_negotiation.cc:407] Connection header received
0925 14:52:54.854126 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.854133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.854212 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.854284 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.857160 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.858231 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.859552 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.860093 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.860239 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.864965 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.864973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.864978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.865014 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.868145 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.868875 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.868881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.868884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.869012 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.869382 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.869387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.869390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.869767 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.869990 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.870337 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.870547 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1094,"thread_start_us":1020,"threads_started":1}
I20260925 14:52:54.877014 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.834768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.17.133.253:39459)
0925 14:52:54.850540 (+ 15772us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.850548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.860698 (+ 10150us) server_negotiation.cc:407] Connection header received
0925 14:52:54.860760 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.860765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.860879 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.860990 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.863022 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.863803 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.865895 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.866341 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.866480 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.869646 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.869655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.869661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.869697 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.873830 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.874491 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.874497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.874500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.874632 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.874957 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.874961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.874965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.875278 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.875506 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.876270 (+ 764us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.876645 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15643,"thread_start_us":24646,"threads_started":1}
I20260925 14:52:54.878576 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.860680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.17.133.252:34371)
0925 14:52:54.860838 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.860845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.861731 (+ 886us) server_negotiation.cc:407] Connection header received
0925 14:52:54.861981 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.861986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.862100 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.862187 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.864989 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.865681 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.867452 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.868159 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.868293 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.871792 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.871798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.871802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.871837 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.875066 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.875770 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.875775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.875778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.875897 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.876334 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.876339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.876342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.876708 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.876980 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.877979 (+ 999us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.878220 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:52:54.875234 24080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:54.883858 24080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:54.930284 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.902570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:41273 (local address 127.0.0.1:49700)
0925 14:52:54.921428 (+ 18858us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.921455 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.921612 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.923114 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.923121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.923155 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:54.923609 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.923624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.925459 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.925466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.927570 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.927584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.929409 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.929415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.929595 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.929622 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.929742 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.929863 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18682,"mutex_wait_us":3,"thread_start_us":15626,"threads_started":1}
I20260925 14:52:54.930981 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.902773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.17.133.194:41273)
0925 14:52:54.914945 (+ 12172us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.914958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.921579 (+ 6621us) server_negotiation.cc:407] Connection header received
0925 14:52:54.921778 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.921798 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.922102 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.922345 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:54.923880 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.925270 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.927758 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.929047 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.929229 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.929445 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.930540 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.930708 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11957,"thread_start_us":11834,"threads_started":1}
I20260925 14:52:54.932775 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.902114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36853 (local address 127.0.0.1:46836)
0925 14:52:54.917868 (+ 15754us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.917895 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.918080 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.925360 (+ 7280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.925364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.925389 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:54.925866 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.925880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.927860 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.927867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.930234 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.930248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.932099 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.932104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.932241 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.932264 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.932340 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.932424 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15573,"mutex_wait_us":38,"thread_start_us":22132,"threads_started":1}
I20260925 14:52:54.933013 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.902383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.17.133.193:36853)
0925 14:52:54.924867 (+ 22484us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.924873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.924894 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:52:54.924972 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.924977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.925159 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.925371 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:54.926770 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.927729 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.930421 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.931896 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.932058 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.932248 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.932397 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.932543 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22328,"thread_start_us":21615,"threads_started":1}
I20260925 14:52:54.935093 25128 tablet_service.cc:1467] Processing CreateTablet for tablet 8665d04691914427b3bedf6628096231 (DEFAULT_TABLE table=test-table [id=a75130e054a64e2cbc11c6d30e712941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:54.936360 25128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8665d04691914427b3bedf6628096231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:54.935993 24983 tablet_service.cc:1467] Processing CreateTablet for tablet 8665d04691914427b3bedf6628096231 (DEFAULT_TABLE table=test-table [id=a75130e054a64e2cbc11c6d30e712941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:54.937194 24983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8665d04691914427b3bedf6628096231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:54.942921 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.903682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:40247 (local address 127.0.0.1:50536)
0925 14:52:54.924478 (+ 20796us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.924500 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.924661 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.926061 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.926067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.926091 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:54.926543 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.926556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.928496 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.928504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.931231 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.931244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.942107 (+ 10863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.942113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.942293 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.942326 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.942417 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.942562 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20689,"mutex_wait_us":2,"thread_start_us":18025,"threads_started":1}
I20260925 14:52:54.943542 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.903314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.17.133.195:40247)
0925 14:52:54.923436 (+ 20122us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.923449 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.924629 (+ 1180us) server_negotiation.cc:407] Connection header received
0925 14:52:54.925140 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.925149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.925417 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.925886 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:54.926727 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.928356 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.931426 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.941587 (+ 10161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.941792 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.941963 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.943112 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.943268 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19875,"thread_start_us":20129,"threads_started":1}
I20260925 14:52:54.946987 25269 tablet_service.cc:1467] Processing CreateTablet for tablet 8665d04691914427b3bedf6628096231 (DEFAULT_TABLE table=test-table [id=a75130e054a64e2cbc11c6d30e712941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:54.948173 25367 tablet_bootstrap.cc:492] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: Bootstrap starting.
I20260925 14:52:54.948307 25269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8665d04691914427b3bedf6628096231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:54.949848 25368 tablet_bootstrap.cc:492] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: Bootstrap starting.
I20260925 14:52:54.952317 25367 tablet_bootstrap.cc:654] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:54.953606 25368 tablet_bootstrap.cc:654] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:54.953616 25367 log.cc:824] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:54.954643 25368 log.cc:824] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:54.956427 25367 tablet_bootstrap.cc:492] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: No bootstrap required, opened a new log
I20260925 14:52:54.956722 25367 ts_tablet_manager.cc:1397] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 14:52:54.957445 25368 tablet_bootstrap.cc:492] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: No bootstrap required, opened a new log
I20260925 14:52:54.957688 25368 ts_tablet_manager.cc:1397] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 14:52:54.961894 25371 tablet_bootstrap.cc:492] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: Bootstrap starting.
I20260925 14:52:54.965111 25371 tablet_bootstrap.cc:654] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:54.966586 25371 log.cc:824] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:54.967695 25367 raft_consensus.cc:348] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:54.968149 25367 raft_consensus.cc:374] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:54.968266 25367 raft_consensus.cc:729] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a38ecf8c754ae68ed0f8d7e2630e21, State: Initialized, Role: FOLLOWER
I20260925 14:52:54.968343 25368 raft_consensus.cc:348] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:54.968745 25368 raft_consensus.cc:374] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:54.968832 25368 raft_consensus.cc:729] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e79e06fdab4fa6a55ee62e07e08231, State: Initialized, Role: FOLLOWER
I20260925 14:52:54.968907 25367 consensus_queue.cc:260] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:54.969630 25371 tablet_bootstrap.cc:492] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: No bootstrap required, opened a new log
I20260925 14:52:54.969472 25368 consensus_queue.cc:260] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:54.970053 25371 ts_tablet_manager.cc:1397] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260925 14:52:54.977135 25371 raft_consensus.cc:348] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:54.977519 25371 raft_consensus.cc:374] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:54.977602 25371 raft_consensus.cc:729] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc52110daacc4bd0bcc641e389ccd486, State: Initialized, Role: FOLLOWER
I20260925 14:52:54.978092 25367 ts_tablet_manager.cc:1428] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: Time spent starting tablet: real 0.021s user 0.019s sys 0.001s
I20260925 14:52:54.978405 25194 heartbeater.cc:503] Master 127.17.133.254:40045 was elected leader, sending a full tablet report...
I20260925 14:52:54.978286 25371 consensus_queue.cc:260] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:54.979864 25368 ts_tablet_manager.cc:1428] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: Time spent starting tablet: real 0.022s user 0.010s sys 0.012s
I20260925 14:52:54.981652 25371 ts_tablet_manager.cc:1428] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20260925 14:52:54.981876 25335 heartbeater.cc:503] Master 127.17.133.254:40045 was elected leader, sending a full tablet report...
W20260925 14:52:55.052783 25337 tablet.cc:2369] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:55.131279 25051 tablet.cc:2369] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:55.213063 25375 raft_consensus.cc:482] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:55.213339 25375 raft_consensus.cc:504] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
W20260925 14:52:55.214838 25196 tablet.cc:2369] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:55.215844 25375 leader_election.cc:288] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853), 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273)
I20260925 14:52:55.221957 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.216272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:49859 (local address 127.17.133.193:36853)
0925 14:52:55.216487 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.216492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.216506 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:52:55.216563 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.216568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.216674 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.216759 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:55.217175 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.218191 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.220231 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.221123 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.221276 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.221347 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.221597 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.221717 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:52:55.221957 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.215509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36853 (local address 127.17.133.195:49859)
0925 14:52:55.215738 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:55.215764 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:55.215874 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:55.216870 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:55.216878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:55.216922 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:55.217045 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.217052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.218337 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.218347 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.220113 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.220121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.221266 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.221272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.221452 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.221474 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:52:55.221573 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:52:55.221696 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 14:52:55.222761 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.216026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:41273 (local address 127.17.133.195:37783)
0925 14:52:55.216214 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:55.216245 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:55.216319 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:55.216722 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:55.216727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:55.216763 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:55.217001 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.217011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.218002 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.218009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.219912 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.219925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.222072 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.222078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.222287 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.222316 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 14:52:55.222397 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:52:55.222504 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":1}
I20260925 14:52:55.222960 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.216200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:37783 (local address 127.17.133.194:41273)
0925 14:52:55.216394 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.216400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.216418 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:52:55.216484 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.216489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.216561 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.216670 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:55.217143 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.217877 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.220068 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.221745 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.221893 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.221970 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.222617 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.222746 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:52:55.222879 25003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8665d04691914427b3bedf6628096231" candidate_uuid: "dc52110daacc4bd0bcc641e389ccd486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e79e06fdab4fa6a55ee62e07e08231" is_pre_election: true
I20260925 14:52:55.223363 25003 raft_consensus.cc:2454] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc52110daacc4bd0bcc641e389ccd486 in term 0.
I20260925 14:52:55.223712 25148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8665d04691914427b3bedf6628096231" candidate_uuid: "dc52110daacc4bd0bcc641e389ccd486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" is_pre_election: true
I20260925 14:52:55.224051 25224 leader_election.cc:302] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e79e06fdab4fa6a55ee62e07e08231, dc52110daacc4bd0bcc641e389ccd486; no voters:
I20260925 14:52:55.224198 25148 raft_consensus.cc:2454] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc52110daacc4bd0bcc641e389ccd486 in term 0.
I20260925 14:52:55.224406 25375 raft_consensus.cc:2781] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:55.224525 25375 raft_consensus.cc:482] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:55.224674 25375 raft_consensus.cc:3037] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:55.226804 25375 raft_consensus.cc:504] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:55.227360 25375 leader_election.cc:288] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [CANDIDATE]: Term 1 election: Requested vote from peers 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853), 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273)
I20260925 14:52:55.227706 25003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8665d04691914427b3bedf6628096231" candidate_uuid: "dc52110daacc4bd0bcc641e389ccd486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e79e06fdab4fa6a55ee62e07e08231"
I20260925 14:52:55.227813 25148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8665d04691914427b3bedf6628096231" candidate_uuid: "dc52110daacc4bd0bcc641e389ccd486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21"
I20260925 14:52:55.227901 25003 raft_consensus.cc:3037] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:55.228011 25148 raft_consensus.cc:3037] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:55.230059 25003 raft_consensus.cc:2454] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc52110daacc4bd0bcc641e389ccd486 in term 1.
I20260925 14:52:55.230202 25148 raft_consensus.cc:2454] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc52110daacc4bd0bcc641e389ccd486 in term 1.
I20260925 14:52:55.230410 25224 leader_election.cc:302] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e79e06fdab4fa6a55ee62e07e08231, dc52110daacc4bd0bcc641e389ccd486; no voters:
I20260925 14:52:55.230667 25375 raft_consensus.cc:2781] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:55.231441 25375 raft_consensus.cc:686] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 1 LEADER]: Becoming Leader. State: Replica: dc52110daacc4bd0bcc641e389ccd486, State: Running, Role: LEADER
I20260925 14:52:55.231897 25375 consensus_queue.cc:237] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:55.236856 24080 catalog_manager.cc:5495] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc52110daacc4bd0bcc641e389ccd486 (127.17.133.195). New cstate: current_term: 1 leader_uuid: "dc52110daacc4bd0bcc641e389ccd486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } health_report { overall_health: HEALTHY } } }
I20260925 14:52:55.296934 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.291918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.17.133.195:40247)
0925 14:52:55.292131 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.292137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.292156 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:52:55.292266 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.292270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.292347 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.292490 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:55.292852 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.293555 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.294789 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.295188 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.295317 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.295752 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:55.296301 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:55.296464 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.296558 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.296656 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:52:55.305356 25148 raft_consensus.cc:1261] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 1 FOLLOWER]: Refusing update from remote peer dc52110daacc4bd0bcc641e389ccd486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:55.305356 25003 raft_consensus.cc:1261] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [term 1 FOLLOWER]: Refusing update from remote peer dc52110daacc4bd0bcc641e389ccd486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:55.306375 25376 consensus_queue.cc:1035] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:55.306669 25375 consensus_queue.cc:1035] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:55.319566 25379 mvcc.cc:205] Tried to move back new op lower bound from 7333265306840973312 to 7333265306553090048. Current Snapshot: MvccSnapshot[applied={T|T < 7333265306840973312}]
I20260925 14:52:55.321002 25380 mvcc.cc:205] Tried to move back new op lower bound from 7333265306840973312 to 7333265306553090048. Current Snapshot: MvccSnapshot[applied={T|T < 7333265306840973312}]
I20260925 14:52:55.337750 24079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:55.338008 24079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:55.345144 24079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1ddf08cf4ea40398ad8a455ebdd9f6b: Sending DeleteTablet for 3 replicas of tablet 8665d04691914427b3bedf6628096231
I20260925 14:52:55.346306 24983 tablet_service.cc:1514] Processing DeleteTablet for tablet 8665d04691914427b3bedf6628096231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:55 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:46836
I20260925 14:52:55.346678 25128 tablet_service.cc:1514] Processing DeleteTablet for tablet 8665d04691914427b3bedf6628096231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:55 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:49700
I20260925 14:52:55.347028 25269 tablet_service.cc:1514] Processing DeleteTablet for tablet 8665d04691914427b3bedf6628096231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:55 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:50536
I20260925 14:52:55.348266 25388 tablet_replica.cc:330] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: stopping tablet replica
I20260925 14:52:55.348807 25388 raft_consensus.cc:2229] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:55.349364 25388 raft_consensus.cc:2258] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:55.351995 25388 ts_tablet_manager.cc:1905] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:55.358618 25390 tablet_replica.cc:330] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: stopping tablet replica
I20260925 14:52:55.359155 25390 raft_consensus.cc:2229] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:55.359839 25390 raft_consensus.cc:2258] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:55.360426 25388 ts_tablet_manager.cc:1918] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:55.360572 25388 log.cc:1196] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/8665d04691914427b3bedf6628096231
I20260925 14:52:55.361166 25388 ts_tablet_manager.cc:1939] T 8665d04691914427b3bedf6628096231 P 03e79e06fdab4fa6a55ee62e07e08231: Deleting consensus metadata
I20260925 14:52:55.362272 25390 ts_tablet_manager.cc:1905] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:55.362948 24066 catalog_manager.cc:4841] TS 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853): tablet 8665d04691914427b3bedf6628096231 (table test-table [id=a75130e054a64e2cbc11c6d30e712941]) successfully deleted
I20260925 14:52:55.368330 25390 ts_tablet_manager.cc:1918] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:55.368517 25390 log.cc:1196] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/8665d04691914427b3bedf6628096231
I20260925 14:52:55.369212 25390 ts_tablet_manager.cc:1939] T 8665d04691914427b3bedf6628096231 P dc52110daacc4bd0bcc641e389ccd486: Deleting consensus metadata
I20260925 14:52:55.370221 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 24048
I20260925 14:52:55.371070 24067 catalog_manager.cc:4841] TS dc52110daacc4bd0bcc641e389ccd486 (127.17.133.195:40247): tablet 8665d04691914427b3bedf6628096231 (table test-table [id=a75130e054a64e2cbc11c6d30e712941]) successfully deleted
I20260925 14:52:55.371558 25389 tablet_replica.cc:330] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: stopping tablet replica
I20260925 14:52:55.372364 24110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:55.372045 25389 raft_consensus.cc:2229] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:55.373685 25389 raft_consensus.cc:2258] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:55.376544 25389 ts_tablet_manager.cc:1905] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:55.384366 25389 ts_tablet_manager.cc:1918] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:55.384538 25389 log.cc:1196] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/8665d04691914427b3bedf6628096231
I20260925 14:52:55.385118 25389 ts_tablet_manager.cc:1939] T 8665d04691914427b3bedf6628096231 P 05a38ecf8c754ae68ed0f8d7e2630e21: Deleting consensus metadata
I20260925 14:52:55.388743 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.348180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.17.133.253:39459)
0925 14:52:55.348364 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.348370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.370888 (+ 22518us) server_negotiation.cc:407] Connection header received
0925 14:52:55.371395 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.371402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.371488 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.371596 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.374410 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.375544 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.377472 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.378106 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.378203 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.382291 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.382297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.382302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.382340 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.385909 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.386807 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.386815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.386821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.386947 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.387307 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.387312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.387316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.387589 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.387889 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.388146 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.388347 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:52:55.401849 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.369482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.17.133.252:34371)
0925 14:52:55.369689 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.369695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.389168 (+ 19473us) server_negotiation.cc:407] Connection header received
0925 14:52:55.389368 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.389374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.389466 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.389537 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.391033 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.391750 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.393011 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.393398 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.393528 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.395817 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.395823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.395831 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.395861 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.398597 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.399165 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.399171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.399175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.399301 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.399661 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.399697 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.399701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.399997 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.400237 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.400525 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.401611 (+ 1086us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 14:52:55.528088 24065 catalog_manager.cc:4841] TS 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273): tablet 8665d04691914427b3bedf6628096231 (table test-table [id=a75130e054a64e2cbc11c6d30e712941]) successfully deleted
I20260925 14:52:55.528769 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 24048
I20260925 14:52:55.578305 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.561380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.17.133.253:39459)
0925 14:52:55.561578 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.561584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.562740 (+ 1156us) server_negotiation.cc:407] Connection header received
0925 14:52:55.562990 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.563001 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.563085 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.563180 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.565080 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.565968 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.567824 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.568463 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.568569 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.571741 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.571747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.571752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.571789 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.575354 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.576119 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.576126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.576130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.576250 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.576679 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.576685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.576691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.577004 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.577233 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.577793 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.577967 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:52:55.579097 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.562730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.17.133.252:34371)
0925 14:52:55.562927 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.562933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.563141 (+ 208us) server_negotiation.cc:407] Connection header received
0925 14:52:55.563313 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.563318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.563410 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.563531 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.565636 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.566476 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.567752 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.568432 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.568570 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.572045 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.572053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.572058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.572098 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.575535 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.576439 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.576444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.576447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.576632 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.576999 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.577032 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.577038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.577437 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.577677 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.578060 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.578246 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:52:56.323462 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.322713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.195:50743)
0925 14:52:56.323204 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.323304 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":277,"threads_started":1}
W20260925 14:52:56.323850 25335 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:40045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
I20260925 14:52:56.325081 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.324324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.193:49029)
0925 14:52:56.324776 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.324902 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":243,"threads_started":1}
W20260925 14:52:56.325553 25049 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:40045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
I20260925 14:52:56.333552 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.332708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.194:44923)
0925 14:52:56.333250 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.333375 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":310,"threads_started":1}
W20260925 14:52:56.333971 25194 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:40045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
I20260925 14:52:56.844383 25402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1ddf08cf4ea40398ad8a455ebdd9f6b)
I20260925 14:52:56.844720 25402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } }
I20260925 14:52:56.845156 25403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1ddf08cf4ea40398ad8a455ebdd9f6b)
I20260925 14:52:56.845389 25403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } }
I20260925 14:52:56.846215 25402 leader_election.cc:288] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1ddf08cf4ea40398ad8a455ebdd9f6b (127.17.133.254:40045), 9726fc4e8392492eb50bbc22256bae0e (127.17.133.252:34371)
I20260925 14:52:56.846688 25403 leader_election.cc:288] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1ddf08cf4ea40398ad8a455ebdd9f6b (127.17.133.254:40045), 703f693915d3422f99a91c1497d7b24c (127.17.133.253:39459)
I20260925 14:52:56.858636 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.846292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57632)
0925 14:52:56.858081 (+ 11789us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.858308 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11617,"threads_started":1}
W20260925 14:52:56.860466 24663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
I20260925 14:52:56.861291 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.847287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57644)
0925 14:52:56.860779 (+ 13492us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.860982 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13371,"thread_start_us":13231,"threads_started":1}
W20260925 14:52:56.861994 24855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
I20260925 14:52:56.863857 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.863446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57656)
0925 14:52:56.863621 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.863712 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 14:52:56.864162 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.863770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57664)
0925 14:52:56.863913 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.864012 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 14:52:56.864357 24663 leader_election.cc:334] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1ddf08cf4ea40398ad8a455ebdd9f6b (127.17.133.254:40045): Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
W20260925 14:52:56.864516 24855 leader_election.cc:334] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1ddf08cf4ea40398ad8a455ebdd9f6b (127.17.133.254:40045): Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
I20260925 14:52:56.867646 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.847133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.17.133.253:39459)
0925 14:52:56.860652 (+ 13519us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.860659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.860678 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:52:56.860741 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.860748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.860833 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.860933 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.861899 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.863233 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.865602 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.866908 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.867043 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.867127 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.867296 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.867412 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":13340,"thread_start_us":13159,"threads_started":1}
I20260925 14:52:56.867820 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.846876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:39459 (local address 127.0.0.1:39648)
0925 14:52:56.859694 (+ 12818us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.859720 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.859882 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.861180 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.861185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.861223 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.861644 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.861661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.863480 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.863488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.865439 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.865450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.867045 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.867049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.867160 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.867183 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.867261 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.867398 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12650,"mutex_wait_us":1,"thread_start_us":13100,"threads_started":1}
I20260925 14:52:56.868490 24688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9726fc4e8392492eb50bbc22256bae0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "703f693915d3422f99a91c1497d7b24c" is_pre_election: true
I20260925 14:52:56.868918 24688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9726fc4e8392492eb50bbc22256bae0e in term 1.
I20260925 14:52:56.869568 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.846686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:34371 (local address 127.0.0.1:39914)
0925 14:52:56.859760 (+ 13074us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.859783 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.859926 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.862238 (+ 2312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.862243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.862268 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.863360 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.863374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.865101 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.865106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.867208 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.867217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.868791 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.868800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.868923 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.868967 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.869079 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.869201 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12918,"mutex_wait_us":67,"thread_start_us":11026,"threads_started":1}
I20260925 14:52:56.869588 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.860808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.17.133.252:34371)
0925 14:52:56.861945 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.861950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.861964 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:52:56.862023 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.862029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.862119 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.862216 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.863573 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.864973 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.867368 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.868572 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.868772 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.868834 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.869100 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.869223 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":1018,"thread_start_us":888,"threads_started":1}
I20260925 14:52:56.869472 24857 leader_election.cc:302] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 703f693915d3422f99a91c1497d7b24c, 9726fc4e8392492eb50bbc22256bae0e; no voters: c1ddf08cf4ea40398ad8a455ebdd9f6b
I20260925 14:52:56.869861 25403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:52:56.869997 25403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1ddf08cf4ea40398ad8a455ebdd9f6b)
I20260925 14:52:56.870083 25403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:52:56.870131 24880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "703f693915d3422f99a91c1497d7b24c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9726fc4e8392492eb50bbc22256bae0e" is_pre_election: true
I20260925 14:52:56.872411 25403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } }
I20260925 14:52:56.872905 24880 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f693915d3422f99a91c1497d7b24c in current term 2: Already voted for candidate 9726fc4e8392492eb50bbc22256bae0e in this term.
I20260925 14:52:56.873081 25403 leader_election.cc:288] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [CANDIDATE]: Term 2 election: Requested vote from peers c1ddf08cf4ea40398ad8a455ebdd9f6b (127.17.133.254:40045), 703f693915d3422f99a91c1497d7b24c (127.17.133.253:39459)
I20260925 14:52:56.873431 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.873136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57666)
0925 14:52:56.873279 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.873337 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:52:56.873505 24688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9726fc4e8392492eb50bbc22256bae0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "703f693915d3422f99a91c1497d7b24c"
I20260925 14:52:56.873710 24688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:52:56.873891 24665 leader_election.cc:302] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 703f693915d3422f99a91c1497d7b24c; no voters: 9726fc4e8392492eb50bbc22256bae0e, c1ddf08cf4ea40398ad8a455ebdd9f6b
I20260925 14:52:56.874449 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.874099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57676)
0925 14:52:56.874271 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.874353 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260925 14:52:56.874650 24855 leader_election.cc:334] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1ddf08cf4ea40398ad8a455ebdd9f6b (127.17.133.254:40045): Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
I20260925 14:52:56.875973 24688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9726fc4e8392492eb50bbc22256bae0e in term 2.
I20260925 14:52:56.876151 25402 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:52:56.876401 24857 leader_election.cc:302] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 703f693915d3422f99a91c1497d7b24c, 9726fc4e8392492eb50bbc22256bae0e; no voters: c1ddf08cf4ea40398ad8a455ebdd9f6b
I20260925 14:52:56.876685 25403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:52:56.877159 25403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [term 2 LEADER]: Becoming Leader. State: Replica: 9726fc4e8392492eb50bbc22256bae0e, State: Running, Role: LEADER
I20260925 14:52:56.877530 25403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } }
I20260925 14:52:56.879251 25412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9726fc4e8392492eb50bbc22256bae0e. Latest consensus state: current_term: 2 leader_uuid: "9726fc4e8392492eb50bbc22256bae0e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:56.879539 25412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:56.880055 25414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:56.883388 25414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:56.884143 25414 catalog_manager.cc:1259] Loaded cluster ID: f6ee1cb7e85e4426a3353ff2e9df5f63
I20260925 14:52:56.884222 25414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:56.885331 25414 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:56.886047 25414 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Loaded TSK: 0
I20260925 14:52:56.886842 25414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:52:56.912628 24868 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:56.919220 24868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:56.923043 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.922687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57696)
0925 14:52:56.922857 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.922938 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:52:56.923135 24688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [term 2 FOLLOWER]: Refusing update from remote peer 9726fc4e8392492eb50bbc22256bae0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 14:52:56.923916 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.923533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57704)
0925 14:52:56.923679 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.923744 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:52:56.923880 25412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 14:52:56.924119 24855 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e -> Peer c1ddf08cf4ea40398ad8a455ebdd9f6b (127.17.133.254:40045): Couldn't send request to peer c1ddf08cf4ea40398ad8a455ebdd9f6b. Status: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:52:56.926347 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9726fc4e8392492eb50bbc22256bae0e. Latest consensus state: current_term: 2 leader_uuid: "9726fc4e8392492eb50bbc22256bae0e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:56.926558 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:52:56.927927 25412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9726fc4e8392492eb50bbc22256bae0e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:56.928193 25412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:56.929071 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9726fc4e8392492eb50bbc22256bae0e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1ddf08cf4ea40398ad8a455ebdd9f6b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "703f693915d3422f99a91c1497d7b24c" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 39459 } attrs { promote: false } } peers { permanent_uuid: "9726fc4e8392492eb50bbc22256bae0e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 34371 } attrs { promote: false } } }
I20260925 14:52:56.929342 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703f693915d3422f99a91c1497d7b24c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:52:56.945603 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.935853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:41273 (local address 127.0.0.1:49704)
0925 14:52:56.935983 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.936000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.936102 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.937087 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.937095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.937125 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.937389 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.937404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.939913 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.939921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.942625 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.942643 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.944663 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.944672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.944914 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.944968 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.945048 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.945224 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:52:56.947146 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.935957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.17.133.194:41273)
0925 14:52:56.936421 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.936429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.936447 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:52:56.936514 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.936520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.936950 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.937551 (+ 601us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.938223 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.939716 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.942883 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.944404 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.946358 (+ 1954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.946483 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.946543 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.946755 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":223,"threads_started":1}
I20260925 14:52:56.947995 25128 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc22bf8a5184f319dafe2d7e348389d (DEFAULT_TABLE table=test-table [id=f247f5476d024e3baf08429fb1940d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:56.948554 25128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc22bf8a5184f319dafe2d7e348389d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:56.952685 25423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.936870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.17.133.195:40247)
0925 14:52:56.947992 (+ 11122us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.947996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.948010 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:52:56.948061 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.948064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.948122 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.948330 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.948795 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.949458 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.950941 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.951763 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.951921 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.951988 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.952258 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.952379 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11028,"thread_start_us":10949,"threads_started":1}
I20260925 14:52:56.952730 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.936533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:40247 (local address 127.0.0.1:50548)
0925 14:52:56.936672 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.936696 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.936788 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.948336 (+ 11548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.948341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.948363 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.948637 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.948648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.949583 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.949591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.950789 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.950800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.951960 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.951965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.952126 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.952146 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.952235 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.952379 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:52:56.953434 25269 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc22bf8a5184f319dafe2d7e348389d (DEFAULT_TABLE table=test-table [id=f247f5476d024e3baf08429fb1940d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:56.953852 25269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc22bf8a5184f319dafe2d7e348389d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:56.956629 25425 tablet_bootstrap.cc:492] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: Bootstrap starting.
I20260925 14:52:56.958451 25425 tablet_bootstrap.cc:654] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:56.961112 25427 tablet_bootstrap.cc:492] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: Bootstrap starting.
I20260925 14:52:56.962626 25427 tablet_bootstrap.cc:654] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:56.964457 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.936609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36853 (local address 127.0.0.1:46848)
0925 14:52:56.945856 (+ 9247us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.945876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.945996 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.957167 (+ 11171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.957175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.957203 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.957399 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.957411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.959033 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.959038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.961556 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.961576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.963631 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.963639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.963882 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.963932 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.964033 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.964180 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9134,"thread_start_us":10761,"threads_started":1}
I20260925 14:52:56.964488 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.936984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.17.133.193:36853)
0925 14:52:56.956879 (+ 19895us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.956885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.956905 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:52:56.956981 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.956985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.957052 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.957149 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.957616 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.958855 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.961829 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.963462 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.963662 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.963817 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.964066 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.964189 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19731,"thread_start_us":20254,"threads_started":1}
I20260925 14:52:56.964915 25425 tablet_bootstrap.cc:492] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: No bootstrap required, opened a new log
I20260925 14:52:56.965067 25425 ts_tablet_manager.cc:1397] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 14:52:56.965371 24983 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc22bf8a5184f319dafe2d7e348389d (DEFAULT_TABLE table=test-table [id=f247f5476d024e3baf08429fb1940d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:56.965620 25427 tablet_bootstrap.cc:492] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: No bootstrap required, opened a new log
I20260925 14:52:56.965752 25427 ts_tablet_manager.cc:1397] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 14:52:56.965782 24983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc22bf8a5184f319dafe2d7e348389d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:56.965859 25425 raft_consensus.cc:348] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:56.966122 25425 raft_consensus.cc:374] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:56.966215 25425 raft_consensus.cc:729] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a38ecf8c754ae68ed0f8d7e2630e21, State: Initialized, Role: FOLLOWER
I20260925 14:52:56.966495 25425 consensus_queue.cc:260] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:56.966531 25427 raft_consensus.cc:348] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:56.966723 25427 raft_consensus.cc:374] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:56.966806 25427 raft_consensus.cc:729] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc52110daacc4bd0bcc641e389ccd486, State: Initialized, Role: FOLLOWER
I20260925 14:52:56.967080 25427 consensus_queue.cc:260] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:56.969374 25427 ts_tablet_manager.cc:1428] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:56.969823 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.968295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.195:44787)
0925 14:52:56.969219 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.969445 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":563,"threads_started":1}
I20260925 14:52:56.970058 25425 ts_tablet_manager.cc:1428] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 14:52:56.973227 25334 heartbeater.cc:503] Master 127.17.133.252:34371 was elected leader, sending a full tablet report...
I20260925 14:52:56.974175 25193 heartbeater.cc:503] Master 127.17.133.252:34371 was elected leader, sending a full tablet report...
I20260925 14:52:56.975392 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.971677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.195:47433)
0925 14:52:56.975196 (+ 3519us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.975272 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3421}
I20260925 14:52:56.977236 25433 tablet_bootstrap.cc:492] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: Bootstrap starting.
I20260925 14:52:56.978554 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.970553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.194:48857)
0925 14:52:56.978228 (+ 7675us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.978374 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7550,"thread_start_us":7438,"threads_started":1}
I20260925 14:52:56.979095 25433 tablet_bootstrap.cc:654] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:56.979454 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.979167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.194:49489)
0925 14:52:56.979289 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.979360 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 14:52:56.980823 25433 tablet_bootstrap.cc:492] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: No bootstrap required, opened a new log
I20260925 14:52:56.980954 25433 ts_tablet_manager.cc:1397] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:52:56.981539 25433 raft_consensus.cc:348] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:56.981716 25433 raft_consensus.cc:374] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:56.981802 25433 raft_consensus.cc:729] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e79e06fdab4fa6a55ee62e07e08231, State: Initialized, Role: FOLLOWER
I20260925 14:52:56.982038 25433 consensus_queue.cc:260] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:56.982684 25433 ts_tablet_manager.cc:1428] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:56.984742 25048 heartbeater.cc:503] Master 127.17.133.252:34371 was elected leader, sending a full tablet report...
I20260925 14:52:56.989377 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.982967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.193:36543)
0925 14:52:56.989049 (+ 6082us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.989180 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5947,"thread_start_us":5838,"threads_started":1}
I20260925 14:52:56.990199 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.989897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.193:35345)
0925 14:52:56.990036 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.990105 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 14:52:57.090713 25430 raft_consensus.cc:482] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:57.090919 25430 raft_consensus.cc:504] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:57.091514 25430 leader_election.cc:288] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273), 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853)
I20260925 14:52:57.092038 25003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc22bf8a5184f319dafe2d7e348389d" candidate_uuid: "dc52110daacc4bd0bcc641e389ccd486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e79e06fdab4fa6a55ee62e07e08231" is_pre_election: true
I20260925 14:52:57.092068 25148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc22bf8a5184f319dafe2d7e348389d" candidate_uuid: "dc52110daacc4bd0bcc641e389ccd486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" is_pre_election: true
I20260925 14:52:57.092249 25003 raft_consensus.cc:2454] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc52110daacc4bd0bcc641e389ccd486 in term 0.
I20260925 14:52:57.092310 25148 raft_consensus.cc:2454] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc52110daacc4bd0bcc641e389ccd486 in term 0.
I20260925 14:52:57.092774 25224 leader_election.cc:302] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e79e06fdab4fa6a55ee62e07e08231, dc52110daacc4bd0bcc641e389ccd486; no voters:
I20260925 14:52:57.093006 25430 raft_consensus.cc:2781] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:57.093149 25430 raft_consensus.cc:482] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:57.093252 25430 raft_consensus.cc:3037] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:57.095178 25430 raft_consensus.cc:504] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:57.095683 25430 leader_election.cc:288] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [CANDIDATE]: Term 1 election: Requested vote from peers 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273), 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853)
I20260925 14:52:57.096122 25148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc22bf8a5184f319dafe2d7e348389d" candidate_uuid: "dc52110daacc4bd0bcc641e389ccd486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21"
I20260925 14:52:57.096153 25003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc22bf8a5184f319dafe2d7e348389d" candidate_uuid: "dc52110daacc4bd0bcc641e389ccd486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e79e06fdab4fa6a55ee62e07e08231"
I20260925 14:52:57.096339 25148 raft_consensus.cc:3037] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:57.096342 25003 raft_consensus.cc:3037] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:57.098281 25148 raft_consensus.cc:2454] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc52110daacc4bd0bcc641e389ccd486 in term 1.
I20260925 14:52:57.098300 25003 raft_consensus.cc:2454] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc52110daacc4bd0bcc641e389ccd486 in term 1.
I20260925 14:52:57.098685 25223 leader_election.cc:302] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a38ecf8c754ae68ed0f8d7e2630e21, dc52110daacc4bd0bcc641e389ccd486; no voters:
I20260925 14:52:57.098981 25430 raft_consensus.cc:2781] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:57.099375 25430 raft_consensus.cc:686] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 1 LEADER]: Becoming Leader. State: Replica: dc52110daacc4bd0bcc641e389ccd486, State: Running, Role: LEADER
I20260925 14:52:57.099656 25430 consensus_queue.cc:237] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } }
I20260925 14:52:57.100426 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.099985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.195:58135)
0925 14:52:57.100149 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.100265 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 14:52:57.105608 24869 catalog_manager.cc:5495] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc52110daacc4bd0bcc641e389ccd486 (127.17.133.195). New cstate: current_term: 1 leader_uuid: "dc52110daacc4bd0bcc641e389ccd486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc52110daacc4bd0bcc641e389ccd486" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 40247 } health_report { overall_health: HEALTHY } } }
I20260925 14:52:57.122102 25423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.118036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.17.133.195:40247)
0925 14:52:57.118197 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.118201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.118217 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:52:57.118373 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.118376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.118435 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.118505 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:57.118777 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.119453 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.120687 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.121086 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.121171 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.121499 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:57.121583 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:57.121644 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.121784 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.121837 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:52:57.124889 25148 raft_consensus.cc:1261] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 1 FOLLOWER]: Refusing update from remote peer dc52110daacc4bd0bcc641e389ccd486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:57.124897 25003 raft_consensus.cc:1261] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [term 1 FOLLOWER]: Refusing update from remote peer dc52110daacc4bd0bcc641e389ccd486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:57.125550 25437 consensus_queue.cc:1035] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e79e06fdab4fa6a55ee62e07e08231" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:57.125906 25430 consensus_queue.cc:1035] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a38ecf8c754ae68ed0f8d7e2630e21" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:57.128458 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.127958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.193:47151)
0925 14:52:57.128181 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.128289 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 14:52:57.128470 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.127961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.194:54891)
0925 14:52:57.128152 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.128293 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:52:57.134485 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.133949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.193:40727)
0925 14:52:57.134193 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.134291 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260925 14:52:57.138386 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.137994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.195:47075)
0925 14:52:57.138151 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.138235 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 14:52:57.141928 24870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:57.142186 24870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:57.147799 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.147460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.195:39559)
0925 14:52:57.147587 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.147661 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 14:52:57.148481 24870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9726fc4e8392492eb50bbc22256bae0e: Sending DeleteTablet for 3 replicas of tablet 5fc22bf8a5184f319dafe2d7e348389d
I20260925 14:52:57.149477 25269 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fc22bf8a5184f319dafe2d7e348389d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='oryx', principal='oryx/127.17.133.252@KRBTEST.COM'} at 127.0.0.1:50548
I20260925 14:52:57.149642 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.149225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.17.133.194:49905)
0925 14:52:57.149377 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.149467 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:52:57.149755 25128 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fc22bf8a5184f319dafe2d7e348389d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='oryx', principal='oryx/127.17.133.252@KRBTEST.COM'} at 127.0.0.1:49704
I20260925 14:52:57.150456 25450 tablet_replica.cc:330] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: stopping tablet replica
I20260925 14:52:57.150502 24983 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fc22bf8a5184f319dafe2d7e348389d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='oryx', principal='oryx/127.17.133.252@KRBTEST.COM'} at 127.0.0.1:46848
I20260925 14:52:57.151053 25449 tablet_replica.cc:330] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: stopping tablet replica
I20260925 14:52:57.151257 25449 raft_consensus.cc:2229] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:57.151391 24982 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fc22bf8a5184f319dafe2d7e348389d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='oryx', principal='oryx/127.17.133.252@KRBTEST.COM'} at 127.0.0.1:46848
I20260925 14:52:57.151873 25449 raft_consensus.cc:2258] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:57.153008 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 24844
I20260925 14:52:57.153343 25449 ts_tablet_manager.cc:1905] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:57.153884 25450 raft_consensus.cc:2229] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:57.154497 25450 raft_consensus.cc:2258] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:57.156203 25451 tablet_replica.cc:330] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: stopping tablet replica
I20260925 14:52:57.156225 25450 ts_tablet_manager.cc:1905] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:57.156379 25451 raft_consensus.cc:2229] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:57.156694 25451 raft_consensus.cc:2258] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:57.158354 25451 ts_tablet_manager.cc:1905] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:57.159617 25449 ts_tablet_manager.cc:1918] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:57.159740 25449 log.cc:1196] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/5fc22bf8a5184f319dafe2d7e348389d
I20260925 14:52:57.160169 25449 ts_tablet_manager.cc:1939] T 5fc22bf8a5184f319dafe2d7e348389d P dc52110daacc4bd0bcc641e389ccd486: Deleting consensus metadata
I20260925 14:52:57.161152 24857 catalog_manager.cc:4841] TS dc52110daacc4bd0bcc641e389ccd486 (127.17.133.195:40247): tablet 5fc22bf8a5184f319dafe2d7e348389d (table test-table [id=f247f5476d024e3baf08429fb1940d60]) successfully deleted
I20260925 14:52:57.161409 25450 ts_tablet_manager.cc:1918] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:57.161545 25450 log.cc:1196] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/5fc22bf8a5184f319dafe2d7e348389d
I20260925 14:52:57.161924 25450 ts_tablet_manager.cc:1939] T 5fc22bf8a5184f319dafe2d7e348389d P 05a38ecf8c754ae68ed0f8d7e2630e21: Deleting consensus metadata
I20260925 14:52:57.162735 24855 catalog_manager.cc:4841] TS 05a38ecf8c754ae68ed0f8d7e2630e21 (127.17.133.194:41273): tablet 5fc22bf8a5184f319dafe2d7e348389d (table test-table [id=f247f5476d024e3baf08429fb1940d60]) successfully deleted
I20260925 14:52:57.163769 25451 ts_tablet_manager.cc:1918] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:57.163898 25451 log.cc:1196] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/5fc22bf8a5184f319dafe2d7e348389d
I20260925 14:52:57.164279 25451 ts_tablet_manager.cc:1939] T 5fc22bf8a5184f319dafe2d7e348389d P 03e79e06fdab4fa6a55ee62e07e08231: Deleting consensus metadata
I20260925 14:52:57.165164 24856 catalog_manager.cc:4841] TS 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853): tablet 5fc22bf8a5184f319dafe2d7e348389d (table test-table [id=f247f5476d024e3baf08429fb1940d60]) successfully deleted
W20260925 14:52:57.165563 24856 catalog_manager.cc:4808] TS 03e79e06fdab4fa6a55ee62e07e08231 (127.17.133.193:36853): delete failed for tablet 5fc22bf8a5184f319dafe2d7e348389d because the tablet was not found. No further retry: Not found: Tablet not found: 5fc22bf8a5184f319dafe2d7e348389d
I20260925 14:52:57.175372 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.154653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.17.133.193:36853)
0925 14:52:57.154854 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.154862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.157287 (+ 2425us) server_negotiation.cc:407] Connection header received
0925 14:52:57.157462 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.157467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.157538 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.157655 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:57.166212 (+ 8557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.166868 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.167950 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.168315 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.168413 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.170753 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:57.170764 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:57.170776 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:57.170802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:57.173483 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:57.173987 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:57.173993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:57.173994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:57.174100 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:57.174379 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:57.174382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:57.174384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:57.174670 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:57.174853 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.175015 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.175149 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:52:57.175915 25043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:57.270021 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 24844
I20260925 14:52:57.286767 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 25060
I20260925 14:52:57.298466 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.287482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.17.133.194:41273)
0925 14:52:57.287661 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.287665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.287681 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:52:57.287838 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.287841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.287901 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.287999 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:57.289260 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.289922 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.291040 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.291409 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.291490 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.293776 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:57.293790 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:57.293805 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:57.293833 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:57.296326 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:57.296976 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:57.296983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:57.296985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:57.297102 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:57.297388 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:57.297391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:57.297393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:57.297679 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:57.297900 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.298068 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.298230 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:52:57.299082 25188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:57.387480 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 25060
I20260925 14:52:57.395649 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.395297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57714)
0925 14:52:57.395458 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.395528 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:52:57.396615 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.396277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:57730)
0925 14:52:57.396409 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.396476 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:52:57.404498 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 25206
I20260925 14:52:57.417289 25423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.405185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.17.133.195:40247)
0925 14:52:57.405343 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.405348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.405396 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:52:57.405615 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.405619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.405693 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.405792 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:57.407098 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.407801 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.409038 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.409418 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.409551 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.411966 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:57.411980 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:57.411996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:57.412026 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:57.414735 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:57.415343 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:57.415350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:57.415352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:57.415491 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:57.415824 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:57.415830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:57.415833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:57.416165 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:57.416457 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.416749 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.416950 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:52:57.417938 25329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:57.532630 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 25206
I20260925 14:52:57.553905 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 24541
I20260925 14:52:57.554831 24708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:57.668046 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 24541
I20260925 14:52:57.688649 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 24642
I20260925 14:52:57.689643 24900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:52:57.807248 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 24642
2026-09-25T14:52:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9320 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:57 dist-test-slave-v5vl krb5kdc[25460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:57 dist-test-slave-v5vl krb5kdc[25460](info): set up 2 sockets
Sep 25 14:52:57 dist-test-slave-v5vl krb5kdc[25460](info): commencing operation
krb5kdc: starting...
W20260925 14:52:58.885660 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:52:59 dist-test-slave-v5vl krb5kdc[25460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.254.keytab.
Entry for principal oryx/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.254.keytab.
I20260925 14:52:59.088070 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:33163
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.254.keytab
--principal=oryx/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:59.397807 25473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:59.398039 25473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:59.398167 25473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:59.405130 25473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:59.405215 25473 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:52:59.405339 25473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:59.405423 25473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:59.405562 25473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:59.405660 25473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:59.414773 25473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:33163
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.254.keytab
--principal=oryx/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:59.415905 25473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:59.417124 25473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:59.428340 25478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.428308 25479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.428375 25481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.429183 25473 server_base.cc:1034] running on GCE node
I20260925 14:52:59.429878 25473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:59.431133 25473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:59.432372 25473 hybrid_clock.cc:648] HybridClock initialized: now 1790347979432311 us; error 38 us; skew 500 ppm
I20260925 14:52:59.432732 25473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:52:59 dist-test-slave-v5vl krb5kdc[25460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:59.440447 25473 init.cc:377] Logged in from keytab as oryx/127.17.133.254@KRBTEST.COM (short username oryx)
I20260925 14:52:59.441844 25473 webserver.cc:460] Webserver started at http://127.17.133.254:38061/ using document root <none> and password file <none>
I20260925 14:52:59.442561 25473 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:59.442684 25473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:59.442984 25473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:59.445892 25473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "d4cb4d48255f4972aaa107c3da5c0668"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-v5vl"
server_key: "d4c2c4cd33a0555d1137eadec4bc725f"
server_key_iv: "a4a5de682dbb1f483ed12f6839f8729b"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.446707 25473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "d4cb4d48255f4972aaa107c3da5c0668"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-v5vl"
server_key: "d4c2c4cd33a0555d1137eadec4bc725f"
server_key_iv: "a4a5de682dbb1f483ed12f6839f8729b"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.451495 25473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 14:52:59.454636 25488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.455849 25473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:59.456029 25473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "d4cb4d48255f4972aaa107c3da5c0668"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-v5vl"
server_key: "d4c2c4cd33a0555d1137eadec4bc725f"
server_key_iv: "a4a5de682dbb1f483ed12f6839f8729b"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.456223 25473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:59.478477 25473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:59.482193 25473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:59.482425 25473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:59.504096 25473 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:33163
I20260925 14:52:59.504136 25549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:33163 every 8 connection(s)
I20260925 14:52:59.505448 25473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:59.507781 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 25473
I20260925 14:52:59.508003 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:52:59.508663 17943 external_mini_cluster.cc:1428] Setting key fee8eee7198a7f773b1dc0f4ee965875
I20260925 14:52:59.509374 25550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:59.524076 25550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668: Bootstrap starting.
Sep 25 14:52:59 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:52:59.528317 25550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:59.529788 25550 log.cc:824] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:59.533308 25550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668: No bootstrap required, opened a new log
I20260925 14:52:59.534355 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.510672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.17.133.254:33163)
0925 14:52:59.511448 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.511460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.511500 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:52:59.513814 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.513838 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.514256 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.514636 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.516447 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.517865 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.519804 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.520560 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.520764 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.525570 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.525583 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.525605 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:59.525678 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.529636 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.530476 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.530486 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.530495 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.530681 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.531166 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.531170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.531173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.531830 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.532111 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.532464 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.532941 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":363,"threads_started":1}
I20260925 14:52:59.541460 25550 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cb4d48255f4972aaa107c3da5c0668" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33163 } }
I20260925 14:52:59.541819 25550 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:59.541891 25550 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cb4d48255f4972aaa107c3da5c0668, State: Initialized, Role: FOLLOWER
I20260925 14:52:59.542392 25550 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cb4d48255f4972aaa107c3da5c0668" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33163 } }
I20260925 14:52:59.542578 25550 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:59.542685 25550 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:59.542798 25550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:59.544916 25550 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cb4d48255f4972aaa107c3da5c0668" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33163 } }
I20260925 14:52:59.545356 25550 leader_election.cc:302] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4cb4d48255f4972aaa107c3da5c0668; no voters:
I20260925 14:52:59.545895 25550 leader_election.cc:288] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:59.545998 25555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:59.547101 25555 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [term 1 LEADER]: Becoming Leader. State: Replica: d4cb4d48255f4972aaa107c3da5c0668, State: Running, Role: LEADER
I20260925 14:52:59.547616 25555 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cb4d48255f4972aaa107c3da5c0668" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33163 } }
I20260925 14:52:59.548004 25550 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:59.550489 25557 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4cb4d48255f4972aaa107c3da5c0668. Latest consensus state: current_term: 1 leader_uuid: "d4cb4d48255f4972aaa107c3da5c0668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cb4d48255f4972aaa107c3da5c0668" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33163 } } }
I20260925 14:52:59.550570 25556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4cb4d48255f4972aaa107c3da5c0668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cb4d48255f4972aaa107c3da5c0668" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33163 } } }
I20260925 14:52:59.550861 25557 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:59.550937 25556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:59.554410 25563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:59.558595 25563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:59.566756 25563 catalog_manager.cc:1347] Generated new cluster ID: 45f9d085ffeb4f45ab55a20afd7ac4bb
I20260925 14:52:59.566875 25563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:59.574914 25563 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:59.576967 25563 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:59.589581 25563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668: Generated new TSK 0
I20260925 14:52:59.590541 25563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab.
Entry for principal oryx/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab.
I20260925 14:52:59.644794 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33163
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab
--principal=oryx/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:59.965848 25576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:59.966089 25576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:59.966260 25576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:59.972918 25576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:59.973001 25576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:59.973196 25576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:52:59.973322 25576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:52:59.982438 25576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab
--principal=oryx/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.133.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:52:59.983748 25576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:59.985085 25576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:59.997995 25584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.998008 25581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.998554 25582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.998934 25576 server_base.cc:1034] running on GCE node
I20260925 14:52:59.999504 25576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:00.000335 25576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:00.001572 25576 hybrid_clock.cc:648] HybridClock initialized: now 1790347980001491 us; error 61 us; skew 500 ppm
I20260925 14:53:00.001875 25576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:00 dist-test-slave-v5vl krb5kdc[25460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347980, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:00.005424 25576 init.cc:377] Logged in from keytab as oryx/127.17.133.193@KRBTEST.COM (short username oryx)
I20260925 14:53:00.006892 25576 webserver.cc:460] Webserver started at http://127.17.133.193:45867/ using document root <none> and password file <none>
I20260925 14:53:00.007550 25576 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:00.007634 25576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:00.007895 25576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:00.010864 25576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "ec434100daf64be49d5d76499a0f4080"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-v5vl"
server_key: "4462498bb103d74124ac464bbf2e8bee"
server_key_iv: "b08d1ce19485d5d6af56b94a721a013d"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.011575 25576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "ec434100daf64be49d5d76499a0f4080"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-v5vl"
server_key: "4462498bb103d74124ac464bbf2e8bee"
server_key_iv: "b08d1ce19485d5d6af56b94a721a013d"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.016350 25576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 14:53:00.019521 25591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:00.020805 25576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:00.020924 25576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "ec434100daf64be49d5d76499a0f4080"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-v5vl"
server_key: "4462498bb103d74124ac464bbf2e8bee"
server_key_iv: "b08d1ce19485d5d6af56b94a721a013d"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.021086 25576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:00.038218 25576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:00.042513 25576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:00.042727 25576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:00.045099 25576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:00.045212 25576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:00.045375 25576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:00.045454 25576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:53:00 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347980, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:53:00.068418 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.046446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33163 (local address 127.17.133.193:52485)
0925 14:53:00.047251 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.047332 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.049577 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.050352 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.050359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.051123 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.051880 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.051898 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.052950 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.052961 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.054704 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.054724 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.055853 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.055862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.056159 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.056936 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:00.056960 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.060277 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.063897 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.063907 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.063929 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.064534 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.065104 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.065112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.065120 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.065467 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.066109 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.066117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.066529 (+ 412us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.066907 (+ 378us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.067351 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":354,"threads_started":1}
I20260925 14:53:00.069437 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.045847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:52485 (local address 127.17.133.254:33163)
0925 14:53:00.046364 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.046370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.047457 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 14:53:00.049871 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.049876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.049956 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.050097 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.052035 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.052794 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.054857 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.055353 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.055480 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.060498 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.060506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.060513 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.060561 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.063704 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.064788 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.064792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.064794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.064966 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.065647 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.065654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.065659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.065976 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.066139 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.068957 (+ 2818us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.069174 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":280,"threads_started":1}
I20260925 14:53:00.090581 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.076032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:48021 (local address 127.17.133.254:33163)
0925 14:53:00.076248 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.076257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.076868 (+ 611us) server_negotiation.cc:407] Connection header received
0925 14:53:00.077127 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.077135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.077230 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.077372 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.078787 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.080039 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.081830 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.082483 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.082623 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.084333 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.084338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.084343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.084381 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.087907 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.088570 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.088577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.088606 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.088728 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.089072 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.089076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.089080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.089449 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.089600 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.090026 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.090244 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:53:00.090687 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.075756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33163 (local address 127.17.133.193:48021)
0925 14:53:00.076764 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.076780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.076967 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.077417 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.077424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.078123 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.078542 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.078553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.080199 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.080204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.081682 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.081696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.082661 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.082667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.082815 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.083511 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:00.083526 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.084171 (+ 645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.088029 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.088032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.088035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.088450 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.088831 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.088838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.088844 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.088962 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.089563 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.089568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.089766 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.089936 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.090246 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":686,"threads_started":1}
I20260925 14:53:00.099510 25576 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:40507
I20260925 14:53:00.099532 25711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:40507 every 8 connection(s)
I20260925 14:53:00.100734 25576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:00.102584 25712 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33163
I20260925 14:53:00.102814 25712 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:00.103413 25712 heartbeater.cc:511] Master 127.17.133.254:33163 requested a full tablet report, sending...
I20260925 14:53:00.105259 25505 ts_manager.cc:194] Registered new tserver with Master: ec434100daf64be49d5d76499a0f4080 (127.17.133.193:40507)
I20260925 14:53:00.105592 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 25576
I20260925 14:53:00.105814 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:00.106405 17943 external_mini_cluster.cc:1428] Setting key 6e4863a19b29fd6b0e866c619504a1c4
I20260925 14:53:00.106799 25505 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:52485
WARNING: no policy specified for oryx/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab.
Entry for principal oryx/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab.
I20260925 14:53:00.151101 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33163
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab
--principal=oryx/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:00.480983 25716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:00.481237 25716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:00.481395 25716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:00.488564 25716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:00.488737 25716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:00.488888 25716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:53:00.489040 25716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:53:00.499676 25716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab
--principal=oryx/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.133.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:00.501009 25716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:00.502295 25716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:00.516067 25722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:00.516112 25721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:00.516561 25724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:00.517728 25716 server_base.cc:1034] running on GCE node
I20260925 14:53:00.518265 25716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:00.519129 25716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:00.520356 25716 hybrid_clock.cc:648] HybridClock initialized: now 1790347980520289 us; error 41 us; skew 500 ppm
I20260925 14:53:00.520711 25716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:00 dist-test-slave-v5vl krb5kdc[25460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347980, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:00.523644 25716 init.cc:377] Logged in from keytab as oryx/127.17.133.194@KRBTEST.COM (short username oryx)
I20260925 14:53:00.525036 25716 webserver.cc:460] Webserver started at http://127.17.133.194:42359/ using document root <none> and password file <none>
I20260925 14:53:00.525664 25716 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:00.525785 25716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:00.526088 25716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:00.528790 25716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "b1221b13e9374b78b54bb9c757e96241"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-v5vl"
server_key: "6b9c4fabdd22a015cb895772aa3b5c16"
server_key_iv: "e059543af313510a8c7a0dbb84efef21"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.529609 25716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "b1221b13e9374b78b54bb9c757e96241"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-v5vl"
server_key: "6b9c4fabdd22a015cb895772aa3b5c16"
server_key_iv: "e059543af313510a8c7a0dbb84efef21"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.534416 25716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 14:53:00.537693 25731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:00.538937 25716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:53:00.539110 25716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "b1221b13e9374b78b54bb9c757e96241"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-v5vl"
server_key: "6b9c4fabdd22a015cb895772aa3b5c16"
server_key_iv: "e059543af313510a8c7a0dbb84efef21"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.539358 25716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:00.564086 25716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:00.567929 25716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:00.568207 25716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:00.570919 25716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:00.571034 25716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:00.571175 25716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:00.571257 25716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:53:00 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347980, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:53:00.594269 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.571496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:59973 (local address 127.17.133.254:33163)
0925 14:53:00.571729 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.571736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.573225 (+ 1489us) server_negotiation.cc:407] Connection header received
0925 14:53:00.575427 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.575435 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.575546 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.575674 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.577972 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.579052 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.581594 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.582133 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.582267 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.587598 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.587603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.587606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.587638 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.590708 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.591881 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.591888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.591892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.592047 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.592644 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.592650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.592653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.592921 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.593076 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.593719 (+ 643us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.593913 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:53:00.595392 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.572275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33163 (local address 127.17.133.194:59973)
0925 14:53:00.573076 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.573114 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.575042 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.575954 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.575962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.576798 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.577776 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.577806 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.579204 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.579210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.581446 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.581461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.582615 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.582625 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.582926 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.583787 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:00.583811 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.587354 (+ 3543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.590847 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.590860 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.590883 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.591706 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.592195 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.592205 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.592209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.592495 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.593060 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.593067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.593384 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.593629 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.594119 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":383,"threads_started":1}
I20260925 14:53:00.618479 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.605269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:48239 (local address 127.17.133.254:33163)
0925 14:53:00.605450 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.605456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.605633 (+ 177us) server_negotiation.cc:407] Connection header received
0925 14:53:00.605808 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.605816 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.605919 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.606050 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.606909 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.608341 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.610097 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.610553 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.610711 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.612481 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.612492 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.612495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.612521 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.615642 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.616480 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.616487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.616491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.616675 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.617104 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.617111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.617114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.617416 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.617569 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.617925 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.618143 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:53:00.618578 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.604959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33163 (local address 127.17.133.194:48239)
0925 14:53:00.605507 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.605529 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.605673 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.606059 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.606062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.606517 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.606779 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.606787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.608475 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.608479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.609936 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.609948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.610732 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.610739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.610900 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.611634 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:00.611662 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.612320 (+ 658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.615758 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.615764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.615770 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.616361 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.616819 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.616825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.616829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.616996 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.617537 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.617541 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.617711 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.617842 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.618155 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":504,"threads_started":1}
I20260925 14:53:00.631919 25716 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:36821
I20260925 14:53:00.631942 25850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:36821 every 8 connection(s)
I20260925 14:53:00.633102 25716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:00.635085 25851 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33163
I20260925 14:53:00.635262 25851 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:00.635838 25851 heartbeater.cc:511] Master 127.17.133.254:33163 requested a full tablet report, sending...
I20260925 14:53:00.637074 25504 ts_manager.cc:194] Registered new tserver with Master: b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821)
I20260925 14:53:00.637916 25504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:59973
I20260925 14:53:00.640949 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 25716
I20260925 14:53:00.641101 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:00.641458 17943 external_mini_cluster.cc:1428] Setting key 41b66581f7088a3fe1a37d588011763c
WARNING: no policy specified for oryx/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab.
Entry for principal oryx/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab.
I20260925 14:53:00.685925 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33163
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab
--principal=oryx/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:00.994874 25855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:00.995141 25855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:00.995294 25855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:01.002542 25855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:01.002636 25855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:01.002807 25855 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:53:01.002918 25855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:53:01.012298 25855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab
--principal=oryx/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.133.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:01.013537 25855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:01.014886 25855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:01.027715 25860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:01.027683 25861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:01.027827 25863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:01.028811 25855 server_base.cc:1034] running on GCE node
I20260925 14:53:01.029474 25855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:01.030364 25855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:01.031569 25855 hybrid_clock.cc:648] HybridClock initialized: now 1790347981031505 us; error 46 us; skew 500 ppm
I20260925 14:53:01.031876 25855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:01 dist-test-slave-v5vl krb5kdc[25460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347981, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:01.034934 25855 init.cc:377] Logged in from keytab as oryx/127.17.133.195@KRBTEST.COM (short username oryx)
I20260925 14:53:01.036271 25855 webserver.cc:460] Webserver started at http://127.17.133.195:38955/ using document root <none> and password file <none>
I20260925 14:53:01.036993 25855 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:01.037073 25855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:01.037407 25855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:01.040145 25855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "55ef238ea91343ff89e63c711f205c22"
format_stamp: "Formatted at 2026-09-25 14:53:01 on dist-test-slave-v5vl"
server_key: "d2056b1583671bc2606a6397e736c8c9"
server_key_iv: "2a5ab3c829f25fc15763751f983797f8"
server_key_version: "encryptionkey@0"
I20260925 14:53:01.040971 25855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "55ef238ea91343ff89e63c711f205c22"
format_stamp: "Formatted at 2026-09-25 14:53:01 on dist-test-slave-v5vl"
server_key: "d2056b1583671bc2606a6397e736c8c9"
server_key_iv: "2a5ab3c829f25fc15763751f983797f8"
server_key_version: "encryptionkey@0"
I20260925 14:53:01.045629 25855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 14:53:01.048740 25870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:01.049968 25855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:53:01.050129 25855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "55ef238ea91343ff89e63c711f205c22"
format_stamp: "Formatted at 2026-09-25 14:53:01 on dist-test-slave-v5vl"
server_key: "d2056b1583671bc2606a6397e736c8c9"
server_key_iv: "2a5ab3c829f25fc15763751f983797f8"
server_key_version: "encryptionkey@0"
I20260925 14:53:01.050357 25855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:01.101541 25855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:01.105526 25855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:01.105782 25855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:01.108179 25855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:01.108296 25855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:01.108426 25855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:01.108492 25855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:01.109743 25712 heartbeater.cc:503] Master 127.17.133.254:33163 was elected leader, sending a full tablet report...
Sep 25 14:53:01 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347981, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:53:01.132457 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.108967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:37525 (local address 127.17.133.254:33163)
0925 14:53:01.109165 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.109174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.110697 (+ 1523us) server_negotiation.cc:407] Connection header received
0925 14:53:01.113072 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.113077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.113152 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.113258 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:01.115550 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.116797 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.119275 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.119917 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.120044 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.124790 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:01.124797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:01.124801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:01.124873 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:01.128442 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.129425 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.129432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.129437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.129606 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.130364 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.130372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.130379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.130779 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:01.130948 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.131632 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.131959 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:53:01.132997 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.109610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33163 (local address 127.17.133.195:37525)
0925 14:53:01.110474 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.110527 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.112805 (+ 2278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.113529 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.113555 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.114396 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:01.115282 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.115304 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.117003 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.117010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.119137 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.119149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.120187 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.120201 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.120364 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.121097 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:01.121135 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:01.124469 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:01.128613 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:01.128625 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:01.128647 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:01.129288 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:01.129782 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:01.129793 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:01.129798 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:01.130193 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:01.130902 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:01.130908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:01.131220 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.131435 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.131800 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":528,"threads_started":1}
I20260925 14:53:01.151499 25855 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:38797
I20260925 14:53:01.151526 25988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:38797 every 8 connection(s)
I20260925 14:53:01.153059 25855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:01.153749 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 25855
I20260925 14:53:01.153951 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:01.154518 17943 external_mini_cluster.cc:1428] Setting key f82f413fa94d31e84a4049bdcd1ce2e3
I20260925 14:53:01.155339 25990 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33163
I20260925 14:53:01.155596 25990 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:01.156466 25990 heartbeater.cc:511] Master 127.17.133.254:33163 requested a full tablet report, sending...
I20260925 14:53:01.157920 25504 ts_manager.cc:194] Registered new tserver with Master: 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797)
I20260925 14:53:01.158861 25504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:37525
I20260925 14:53:01.167104 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.152338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:58171 (local address 127.17.133.254:33163)
0925 14:53:01.152540 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.152548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.153165 (+ 617us) server_negotiation.cc:407] Connection header received
0925 14:53:01.153374 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.153379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.153486 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.153640 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:01.155661 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.157187 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.159916 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.160561 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.160709 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.162401 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:01.162405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:01.162409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:01.162437 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:01.165117 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.165656 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.165660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.165662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.165774 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.166051 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.166054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.166055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.166301 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:01.166402 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.166726 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.166874 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:53:01.167127 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.152387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33163 (local address 127.17.133.195:58171)
0925 14:53:01.153044 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.153064 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.153219 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.153907 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.153916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.154845 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:01.155443 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.155461 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.157421 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.157430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.159679 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.159699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.160754 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.160759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.160891 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.161518 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:01.161534 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:01.162226 (+ 692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:01.165219 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:01.165223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:01.165225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:01.165565 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:01.165867 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:01.165870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:01.165872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:01.165973 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:01.166396 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:01.166399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:01.166532 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.166636 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.166839 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":372,"threads_started":1}
I20260925 14:53:01.170002 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:53:01 dist-test-slave-v5vl krb5kdc[25460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347981, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:53:01 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:01 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:53:01.215315 25600 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:01.206242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.17.133.254:33163)
0925 14:53:01.206413 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.206418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.206672 (+ 254us) server_negotiation.cc:407] Connection header received
0925 14:53:01.206817 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.206821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.206881 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.206969 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:01.208163 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.208939 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.210204 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.210578 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.210686 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.215111 (+ 4425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54686: BlockingRecv error: recv got EOF from 127.0.0.1:54686 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 14:53:01 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347981, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:53:01.230541 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.217989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.17.133.254:33163)
0925 14:53:01.218198 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.218204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.218426 (+ 222us) server_negotiation.cc:407] Connection header received
0925 14:53:01.218539 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.218542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.218598 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.218675 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:01.219705 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.220374 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.221756 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.222160 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.222247 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.225797 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:01.225802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:01.225805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:01.225835 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:01.228487 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.229115 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.229119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.229121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.229238 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.229543 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.229546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.229548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.229807 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:01.230012 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.230178 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.230321 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:53:01.234476 25504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:01.241214 25504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:01.273041 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.252692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.17.133.193:40507)
0925 14:53:01.265448 (+ 12756us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.265455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.265480 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:53:01.265556 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.265561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.265746 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.265953 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:01.266499 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.268105 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.270576 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.271723 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.271906 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.272083 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.272553 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.272735 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12616,"thread_start_us":12507,"threads_started":1}
I20260925 14:53:01.273142 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.252503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:40507 (local address 127.0.0.1:41040)
0925 14:53:01.263814 (+ 11311us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.263880 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.264106 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.265958 (+ 1852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.265966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.265995 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:01.266345 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.266355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.268214 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.268220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.270425 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.270441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.271921 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.271927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.272190 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.272325 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.272510 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.272716 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10926,"mutex_wait_us":50,"thread_start_us":11825,"threads_started":1}
I20260925 14:53:01.275713 26010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.252970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:38797 (local address 127.0.0.1:40372)
0925 14:53:01.265807 (+ 12837us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.265825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.265945 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.266611 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.266617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.266642 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:01.267002 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.267013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.270026 (+ 3013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.270032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.272297 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.272311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.274900 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.274907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.275064 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.275094 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.275195 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.275327 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12711,"mutex_wait_us":46,"thread_start_us":12583,"threads_started":1}
I20260925 14:53:01.275724 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.253010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.17.133.195:38797)
0925 14:53:01.263960 (+ 10950us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.263973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.266071 (+ 2098us) server_negotiation.cc:407] Connection header received
0925 14:53:01.266189 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.266195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.266410 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.266622 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:01.267230 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.269899 (+ 2669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.272922 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.274436 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.274603 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.274753 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.275257 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.275425 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10725,"thread_start_us":10990,"threads_started":1}
I20260925 14:53:01.276736 25642 tablet_service.cc:1467] Processing CreateTablet for tablet 548ebc49dc2547c28092bade47612810 (DEFAULT_TABLE table=test-table [id=729f4506267d498b945403a91d9a85b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:01.277984 25642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548ebc49dc2547c28092bade47612810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.278376 26013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.267098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:36821 (local address 127.0.0.1:53892)
0925 14:53:01.269035 (+ 1937us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.269050 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.269170 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.269927 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.269931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.269984 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:01.270463 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.270475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.272748 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.272754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.275325 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.275337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.277598 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.277604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.277763 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.277804 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.277918 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.278039 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1799,"thread_start_us":1632,"threads_started":1}
I20260925 14:53:01.278414 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.266243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.17.133.194:36821)
0925 14:53:01.268578 (+ 2335us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.268609 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.269153 (+ 544us) server_negotiation.cc:407] Connection header received
0925 14:53:01.269359 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.269367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.269581 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.269771 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:01.270818 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.272563 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.275487 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.277406 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.277585 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.277754 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.277971 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.278128 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2190,"thread_start_us":2074,"threads_started":1}
I20260925 14:53:01.281836 25781 tablet_service.cc:1467] Processing CreateTablet for tablet 548ebc49dc2547c28092bade47612810 (DEFAULT_TABLE table=test-table [id=729f4506267d498b945403a91d9a85b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:01.282328 25920 tablet_service.cc:1467] Processing CreateTablet for tablet 548ebc49dc2547c28092bade47612810 (DEFAULT_TABLE table=test-table [id=729f4506267d498b945403a91d9a85b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:01.283216 25781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548ebc49dc2547c28092bade47612810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.283531 25920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548ebc49dc2547c28092bade47612810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.291940 26014 tablet_bootstrap.cc:492] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: Bootstrap starting.
I20260925 14:53:01.294287 26015 tablet_bootstrap.cc:492] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: Bootstrap starting.
I20260925 14:53:01.296344 26014 tablet_bootstrap.cc:654] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.296635 26016 tablet_bootstrap.cc:492] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: Bootstrap starting.
I20260925 14:53:01.297514 26015 tablet_bootstrap.cc:654] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.298295 26014 log.cc:824] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:01.298581 26015 log.cc:824] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:01.300058 26016 tablet_bootstrap.cc:654] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.301244 26014 tablet_bootstrap.cc:492] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: No bootstrap required, opened a new log
I20260925 14:53:01.301441 26016 log.cc:824] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:01.301501 26014 ts_tablet_manager.cc:1397] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 14:53:01.302340 26015 tablet_bootstrap.cc:492] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: No bootstrap required, opened a new log
I20260925 14:53:01.302634 26015 ts_tablet_manager.cc:1397] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 14:53:01.304250 26016 tablet_bootstrap.cc:492] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: No bootstrap required, opened a new log
I20260925 14:53:01.304486 26016 ts_tablet_manager.cc:1397] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 14:53:01.308570 26014 raft_consensus.cc:348] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } }
I20260925 14:53:01.309047 26014 raft_consensus.cc:374] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.309147 26014 raft_consensus.cc:729] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec434100daf64be49d5d76499a0f4080, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.309794 26014 consensus_queue.cc:260] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } }
I20260925 14:53:01.311395 26014 ts_tablet_manager.cc:1428] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 14:53:01.313469 26015 raft_consensus.cc:348] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } }
I20260925 14:53:01.313864 26015 raft_consensus.cc:374] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.313956 26015 raft_consensus.cc:729] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ef238ea91343ff89e63c711f205c22, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.314628 26015 consensus_queue.cc:260] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } }
I20260925 14:53:01.314627 26016 raft_consensus.cc:348] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } }
I20260925 14:53:01.315026 26016 raft_consensus.cc:374] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.315142 26016 raft_consensus.cc:729] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1221b13e9374b78b54bb9c757e96241, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.315820 25990 heartbeater.cc:503] Master 127.17.133.254:33163 was elected leader, sending a full tablet report...
I20260925 14:53:01.316401 26015 ts_tablet_manager.cc:1428] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260925 14:53:01.316848 26016 consensus_queue.cc:260] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } }
I20260925 14:53:01.317826 25851 heartbeater.cc:503] Master 127.17.133.254:33163 was elected leader, sending a full tablet report...
I20260925 14:53:01.318388 26016 ts_tablet_manager.cc:1428] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260925 14:53:01.353116 25713 tablet.cc:2369] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:53:01.385078 25852 tablet.cc:2369] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:53:01.410897 25991 tablet.cc:2369] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:01.469696 26022 raft_consensus.cc:482] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:01.470101 26022 raft_consensus.cc:504] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } }
I20260925 14:53:01.472054 26022 leader_election.cc:288] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec434100daf64be49d5d76499a0f4080 (127.17.133.193:40507), 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797)
I20260925 14:53:01.481208 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.472563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:42817 (local address 127.17.133.195:38797)
0925 14:53:01.472784 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.472791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.473422 (+ 631us) server_negotiation.cc:407] Connection header received
0925 14:53:01.473640 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.473645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.473738 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.473828 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:01.474631 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.475734 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.478769 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.479979 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.480143 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.480216 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.480832 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.480960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 14:53:01.481208 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.472465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:40683 (local address 127.17.133.193:40507)
0925 14:53:01.472707 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.472713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.473522 (+ 809us) server_negotiation.cc:407] Connection header received
0925 14:53:01.473650 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.473655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.473723 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.473828 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:01.474662 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.475726 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.478769 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.479979 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.480143 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.480216 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.480832 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.480960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:53:01.481509 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.472498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:38797 (local address 127.17.133.194:42817)
0925 14:53:01.473296 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.473317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.473483 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.473866 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.473871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.473922 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:01.474446 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.474468 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.475868 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.475873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.478602 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.478617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.480250 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.480257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.480611 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.480697 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.480793 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.480993 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":315,"spinlock_wait_cycles":29952,"thread_start_us":482,"threads_started":1}
I20260925 14:53:01.481532 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.472526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:40507 (local address 127.17.133.194:40683)
0925 14:53:01.473422 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.473436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.473532 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.473866 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.473871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.473923 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:01.474503 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.474516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.475868 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.475873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.478602 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.478617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.480253 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.480257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.480614 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.480693 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.480793 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.480993 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":399,"spinlock_wait_cycles":8192,"thread_start_us":389,"threads_started":1}
I20260925 14:53:01.482720 25667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548ebc49dc2547c28092bade47612810" candidate_uuid: "b1221b13e9374b78b54bb9c757e96241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec434100daf64be49d5d76499a0f4080" is_pre_election: true
I20260925 14:53:01.482717 25940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548ebc49dc2547c28092bade47612810" candidate_uuid: "b1221b13e9374b78b54bb9c757e96241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ef238ea91343ff89e63c711f205c22" is_pre_election: true
I20260925 14:53:01.483287 25667 raft_consensus.cc:2454] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1221b13e9374b78b54bb9c757e96241 in term 0.
I20260925 14:53:01.483287 25940 raft_consensus.cc:2454] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1221b13e9374b78b54bb9c757e96241 in term 0.
I20260925 14:53:01.483973 25734 leader_election.cc:302] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55ef238ea91343ff89e63c711f205c22, b1221b13e9374b78b54bb9c757e96241, ec434100daf64be49d5d76499a0f4080; no voters:
I20260925 14:53:01.484407 26022 raft_consensus.cc:2781] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:01.484563 26022 raft_consensus.cc:482] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:01.484719 26022 raft_consensus.cc:3037] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:01.486770 26022 raft_consensus.cc:504] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } }
I20260925 14:53:01.487329 26022 leader_election.cc:288] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [CANDIDATE]: Term 1 election: Requested vote from peers ec434100daf64be49d5d76499a0f4080 (127.17.133.193:40507), 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797)
I20260925 14:53:01.487743 25667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548ebc49dc2547c28092bade47612810" candidate_uuid: "b1221b13e9374b78b54bb9c757e96241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec434100daf64be49d5d76499a0f4080"
I20260925 14:53:01.487743 25940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548ebc49dc2547c28092bade47612810" candidate_uuid: "b1221b13e9374b78b54bb9c757e96241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ef238ea91343ff89e63c711f205c22"
I20260925 14:53:01.487941 25940 raft_consensus.cc:3037] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:01.487941 25667 raft_consensus.cc:3037] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:01.491070 25667 raft_consensus.cc:2454] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1221b13e9374b78b54bb9c757e96241 in term 1.
I20260925 14:53:01.491070 25940 raft_consensus.cc:2454] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1221b13e9374b78b54bb9c757e96241 in term 1.
I20260925 14:53:01.491482 25734 leader_election.cc:302] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55ef238ea91343ff89e63c711f205c22, b1221b13e9374b78b54bb9c757e96241, ec434100daf64be49d5d76499a0f4080; no voters:
I20260925 14:53:01.491725 26022 raft_consensus.cc:2781] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:01.492230 26022 raft_consensus.cc:686] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 1 LEADER]: Becoming Leader. State: Replica: b1221b13e9374b78b54bb9c757e96241, State: Running, Role: LEADER
I20260925 14:53:01.492808 26022 consensus_queue.cc:237] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } }
I20260925 14:53:01.497193 25504 catalog_manager.cc:5495] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1221b13e9374b78b54bb9c757e96241 (127.17.133.194). New cstate: current_term: 1 leader_uuid: "b1221b13e9374b78b54bb9c757e96241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } health_report { overall_health: HEALTHY } } }
I20260925 14:53:01.517257 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.511067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.17.133.194:36821)
0925 14:53:01.511242 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.511250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.511271 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:53:01.511508 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.511512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.511585 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.511703 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:01.512069 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.513139 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.514993 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.515444 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.515533 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.515995 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:01.516628 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:01.516750 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.516910 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.516972 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:53:01.524950 25667 raft_consensus.cc:1261] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Refusing update from remote peer b1221b13e9374b78b54bb9c757e96241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:01.525020 25940 raft_consensus.cc:1261] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Refusing update from remote peer b1221b13e9374b78b54bb9c757e96241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:01.525863 26025 consensus_queue.cc:1035] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:01.526173 26022 consensus_queue.cc:1035] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:01.544927 26030 mvcc.cc:205] Tried to move back new op lower bound from 7333265332316323840 to 7333265332198600704. Current Snapshot: MvccSnapshot[applied={T|T < 7333265332316323840}]
I20260925 14:53:01.550719 25505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:01.551000 25505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:01.554329 25505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668: Sending DeleteTablet for 3 replicas of tablet 548ebc49dc2547c28092bade47612810
I20260925 14:53:01.555339 25642 tablet_service.cc:1514] Processing DeleteTablet for tablet 548ebc49dc2547c28092bade47612810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:01 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:41040
I20260925 14:53:01.555620 25781 tablet_service.cc:1514] Processing DeleteTablet for tablet 548ebc49dc2547c28092bade47612810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:01 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:53892
I20260925 14:53:01.555868 25920 tablet_service.cc:1514] Processing DeleteTablet for tablet 548ebc49dc2547c28092bade47612810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:01 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:40372
I20260925 14:53:01.557780 25505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:01.558760 25505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:01.568993 25641 tablet_service.cc:1467] Processing CreateTablet for tablet 088b9f01c3d848e7b28761a1c2185c86 (DEFAULT_TABLE table=test-table [id=67a1570734984e85826f44f06fa99b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:01.569625 25641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088b9f01c3d848e7b28761a1c2185c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.573408 25780 tablet_service.cc:1467] Processing CreateTablet for tablet 088b9f01c3d848e7b28761a1c2185c86 (DEFAULT_TABLE table=test-table [id=67a1570734984e85826f44f06fa99b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:01.574023 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088b9f01c3d848e7b28761a1c2185c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.577268 25919 tablet_service.cc:1467] Processing CreateTablet for tablet 088b9f01c3d848e7b28761a1c2185c86 (DEFAULT_TABLE table=test-table [id=67a1570734984e85826f44f06fa99b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:01.577804 25919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088b9f01c3d848e7b28761a1c2185c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.578282 26039 tablet_replica.cc:330] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: stopping tablet replica
I20260925 14:53:01.578922 26039 raft_consensus.cc:2229] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:01.579759 26039 raft_consensus.cc:2258] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:01.582461 26039 ts_tablet_manager.cc:1905] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:01.592485 26039 ts_tablet_manager.cc:1918] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:01.592725 26039 log.cc:1196] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/548ebc49dc2547c28092bade47612810
I20260925 14:53:01.593412 26039 ts_tablet_manager.cc:1939] T 548ebc49dc2547c28092bade47612810 P b1221b13e9374b78b54bb9c757e96241: Deleting consensus metadata
I20260925 14:53:01.596251 25491 catalog_manager.cc:4841] TS b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821): tablet 548ebc49dc2547c28092bade47612810 (table test-table [id=729f4506267d498b945403a91d9a85b5]) successfully deleted
I20260925 14:53:01.599510 26016 tablet_bootstrap.cc:492] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: Bootstrap starting.
I20260925 14:53:01.599844 26015 tablet_bootstrap.cc:492] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: Bootstrap starting.
I20260925 14:53:01.601567 26015 tablet_bootstrap.cc:654] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.601821 26016 tablet_bootstrap.cc:654] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.607457 26014 tablet_bootstrap.cc:492] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: Bootstrap starting.
I20260925 14:53:01.609202 26014 tablet_bootstrap.cc:654] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.614822 26016 tablet_bootstrap.cc:492] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: No bootstrap required, opened a new log
I20260925 14:53:01.615031 26016 ts_tablet_manager.cc:1397] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260925 14:53:01.616225 26016 raft_consensus.cc:348] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.616569 26016 raft_consensus.cc:374] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.616722 26016 raft_consensus.cc:729] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1221b13e9374b78b54bb9c757e96241, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.617029 26016 consensus_queue.cc:260] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.620546 26016 ts_tablet_manager.cc:1428] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260925 14:53:01.630523 26015 tablet_bootstrap.cc:492] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: No bootstrap required, opened a new log
I20260925 14:53:01.630671 26040 tablet_replica.cc:330] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: stopping tablet replica
I20260925 14:53:01.630710 26015 ts_tablet_manager.cc:1397] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.004s
I20260925 14:53:01.631294 26040 raft_consensus.cc:2229] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:01.631904 26040 raft_consensus.cc:2258] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:01.632794 26038 tablet_replica.cc:330] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: stopping tablet replica
I20260925 14:53:01.633030 26014 tablet_bootstrap.cc:492] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: No bootstrap required, opened a new log
I20260925 14:53:01.633215 26014 ts_tablet_manager.cc:1397] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.009s
I20260925 14:53:01.633244 26038 raft_consensus.cc:2229] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:01.633652 26038 raft_consensus.cc:2258] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:01.635252 26038 ts_tablet_manager.cc:1905] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:01.635568 26014 raft_consensus.cc:348] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.635805 26014 raft_consensus.cc:374] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.636006 26040 ts_tablet_manager.cc:1905] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:01.637048 26015 raft_consensus.cc:348] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.637315 26014 raft_consensus.cc:729] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec434100daf64be49d5d76499a0f4080, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.637639 26014 consensus_queue.cc:260] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.637951 26015 raft_consensus.cc:374] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.638413 26014 ts_tablet_manager.cc:1428] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260925 14:53:01.638567 26015 raft_consensus.cc:729] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ef238ea91343ff89e63c711f205c22, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.638868 26015 consensus_queue.cc:260] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.639636 26015 ts_tablet_manager.cc:1428] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260925 14:53:01.643546 26038 ts_tablet_manager.cc:1918] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:01.643558 26040 ts_tablet_manager.cc:1918] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:01.643702 26038 log.cc:1196] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/548ebc49dc2547c28092bade47612810
I20260925 14:53:01.643806 26040 log.cc:1196] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/548ebc49dc2547c28092bade47612810
I20260925 14:53:01.644218 26038 ts_tablet_manager.cc:1939] T 548ebc49dc2547c28092bade47612810 P ec434100daf64be49d5d76499a0f4080: Deleting consensus metadata
I20260925 14:53:01.644352 26040 ts_tablet_manager.cc:1939] T 548ebc49dc2547c28092bade47612810 P 55ef238ea91343ff89e63c711f205c22: Deleting consensus metadata
I20260925 14:53:01.645929 25492 catalog_manager.cc:4841] TS ec434100daf64be49d5d76499a0f4080 (127.17.133.193:40507): tablet 548ebc49dc2547c28092bade47612810 (table test-table [id=729f4506267d498b945403a91d9a85b5]) successfully deleted
I20260925 14:53:01.645929 25491 catalog_manager.cc:4841] TS 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797): tablet 548ebc49dc2547c28092bade47612810 (table test-table [id=729f4506267d498b945403a91d9a85b5]) successfully deleted
I20260925 14:53:01.801375 26020 raft_consensus.cc:482] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:01.801594 26020 raft_consensus.cc:504] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.803133 26020 leader_election.cc:288] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821), 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797)
I20260925 14:53:01.810427 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.804484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:49987 (local address 127.17.133.195:38797)
0925 14:53:01.804682 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.804688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.805289 (+ 601us) server_negotiation.cc:407] Connection header received
0925 14:53:01.805461 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.805466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.805546 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.805631 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:01.806122 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.806822 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.808606 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.809463 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.809613 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.809690 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.810029 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.810166 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:53:01.810482 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.804359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:38797 (local address 127.17.133.193:49987)
0925 14:53:01.805146 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.805168 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.805320 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.805660 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.805664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.805688 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:01.805974 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.805982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.806944 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.806952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.808428 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.808440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.809672 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.809678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.809849 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.809884 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.809984 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.810126 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":460,"threads_started":1}
I20260925 14:53:01.811120 25940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088b9f01c3d848e7b28761a1c2185c86" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ef238ea91343ff89e63c711f205c22" is_pre_election: true
I20260925 14:53:01.811398 25940 raft_consensus.cc:2454] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 0.
I20260925 14:53:01.811964 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.803489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:60987 (local address 127.17.133.194:36821)
0925 14:53:01.803725 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.803736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.804283 (+ 547us) server_negotiation.cc:407] Connection header received
0925 14:53:01.804624 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.804634 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.804737 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.804831 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:01.805637 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.806828 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.809409 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.810719 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.810878 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.810949 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.811508 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.811657 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:53:01.811925 25594 leader_election.cc:302] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ef238ea91343ff89e63c711f205c22, ec434100daf64be49d5d76499a0f4080; no voters:
I20260925 14:53:01.812095 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.803485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:36821 (local address 127.17.133.193:60987)
0925 14:53:01.804188 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.804209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.804396 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.805043 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.805046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.805073 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:01.805458 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.805470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.807010 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.807018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.809244 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.809258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.811088 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.811096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.811267 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.811367 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.811466 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.811674 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":417,"threads_started":1}
I20260925 14:53:01.812306 26020 raft_consensus.cc:2781] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:01.812521 26020 raft_consensus.cc:482] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:01.812645 26020 raft_consensus.cc:3037] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:01.813037 25802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088b9f01c3d848e7b28761a1c2185c86" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1221b13e9374b78b54bb9c757e96241" is_pre_election: true
I20260925 14:53:01.813467 25802 raft_consensus.cc:2454] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 0.
I20260925 14:53:01.815205 26020 raft_consensus.cc:504] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.815841 26020 leader_election.cc:288] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 election: Requested vote from peers b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821), 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797)
I20260925 14:53:01.816241 25802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088b9f01c3d848e7b28761a1c2185c86" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1221b13e9374b78b54bb9c757e96241"
I20260925 14:53:01.816329 25940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088b9f01c3d848e7b28761a1c2185c86" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ef238ea91343ff89e63c711f205c22"
I20260925 14:53:01.816421 25802 raft_consensus.cc:3037] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:01.816548 25940 raft_consensus.cc:3037] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:01.818660 25802 raft_consensus.cc:2454] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 1.
I20260925 14:53:01.819059 25594 leader_election.cc:302] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1221b13e9374b78b54bb9c757e96241, ec434100daf64be49d5d76499a0f4080; no voters:
I20260925 14:53:01.819321 26020 raft_consensus.cc:2781] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:01.819365 25940 raft_consensus.cc:2454] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 1.
I20260925 14:53:01.823635 26020 raft_consensus.cc:686] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 1 LEADER]: Becoming Leader. State: Replica: ec434100daf64be49d5d76499a0f4080, State: Running, Role: LEADER
I20260925 14:53:01.824352 26020 consensus_queue.cc:237] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.827986 25505 catalog_manager.cc:5495] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec434100daf64be49d5d76499a0f4080 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "ec434100daf64be49d5d76499a0f4080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } health_report { overall_health: UNKNOWN } } }
Sep 25 14:53:01 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.254@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:53:01.925498 26053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.911274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.17.133.254:33163)
0925 14:53:01.911945 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.911953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.912385 (+ 432us) server_negotiation.cc:407] Connection header received
0925 14:53:01.912473 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.912484 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.912638 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.912741 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:01.914064 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.914894 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.916158 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.916690 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.916836 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.920231 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:01.920236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:01.920239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:01.920264 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:01.922813 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.923482 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.923488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.923492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.923605 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.923951 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.923957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.923962 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.924206 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:01.924379 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.924781 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.925121 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":503,"thread_start_us":340,"threads_started":1}
I20260925 14:53:01.925508 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.911258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33163 (local address 127.0.0.1:54702)
0925 14:53:01.912159 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:01.912178 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:01.912333 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:01.912839 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:01.912853 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:01.913741 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:01.913933 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.913942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.915028 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.915033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.916034 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:01.916045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.916839 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.916852 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:01.917085 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.917756 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:01.917773 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:01.920047 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:01.922957 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:01.922970 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:01.922978 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:01.923361 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:01.923713 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:01.923717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:01.923721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:01.923858 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:01.924370 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:01.924378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:01.924529 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 14:53:01.924711 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 14:53:01.925008 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":42,"spinlock_wait_cycles":3840,"thread_start_us":552,"threads_started":1}
I20260925 14:53:01.931483 25505 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:54702:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:53:01.932770 25505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:01.939517 25642 tablet_service.cc:1467] Processing CreateTablet for tablet 6f9646f0fdd74f328b620f59a6e06445 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f1a47a37f5b149ee9b48dd8f89871b34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:53:01.939965 25780 tablet_service.cc:1467] Processing CreateTablet for tablet 6f9646f0fdd74f328b620f59a6e06445 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f1a47a37f5b149ee9b48dd8f89871b34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:53:01.940104 25642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f9646f0fdd74f328b620f59a6e06445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.939925 25920 tablet_service.cc:1467] Processing CreateTablet for tablet 6f9646f0fdd74f328b620f59a6e06445 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f1a47a37f5b149ee9b48dd8f89871b34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:53:01.940398 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f9646f0fdd74f328b620f59a6e06445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.940419 25920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f9646f0fdd74f328b620f59a6e06445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.947271 26014 tablet_bootstrap.cc:492] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: Bootstrap starting.
I20260925 14:53:01.947364 26015 tablet_bootstrap.cc:492] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: Bootstrap starting.
I20260925 14:53:01.947623 26016 tablet_bootstrap.cc:492] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: Bootstrap starting.
I20260925 14:53:01.949024 26014 tablet_bootstrap.cc:654] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.949075 26015 tablet_bootstrap.cc:654] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.949314 26016 tablet_bootstrap.cc:654] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.958825 26016 tablet_bootstrap.cc:492] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: No bootstrap required, opened a new log
I20260925 14:53:01.959050 26016 ts_tablet_manager.cc:1397] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 14:53:01.959867 26016 raft_consensus.cc:348] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.960052 26014 tablet_bootstrap.cc:492] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: No bootstrap required, opened a new log
I20260925 14:53:01.960130 26016 raft_consensus.cc:374] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.960215 26014 ts_tablet_manager.cc:1397] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260925 14:53:01.960237 26016 raft_consensus.cc:729] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1221b13e9374b78b54bb9c757e96241, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.960553 26016 consensus_queue.cc:260] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.960964 26014 raft_consensus.cc:348] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.961206 26014 raft_consensus.cc:374] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.961084 26025 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:01.961273 26014 raft_consensus.cc:729] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec434100daf64be49d5d76499a0f4080, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.961396 26025 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:01.961541 26014 consensus_queue.cc:260] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.961978 26020 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:01.962276 26020 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:01.962940 26014 ts_tablet_manager.cc:1428] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:53:01.963541 26016 ts_tablet_manager.cc:1428] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:53:01.965320 26015 tablet_bootstrap.cc:492] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: No bootstrap required, opened a new log
I20260925 14:53:01.965577 26015 ts_tablet_manager.cc:1397] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20260925 14:53:01.966312 26015 raft_consensus.cc:348] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.966516 26015 raft_consensus.cc:374] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.966609 26015 raft_consensus.cc:729] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ef238ea91343ff89e63c711f205c22, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.966830 26015 consensus_queue.cc:260] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:01.967334 26015 ts_tablet_manager.cc:1428] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 14:53:01.967236 26021 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:01.967537 26021 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:02.172469 26021 raft_consensus.cc:482] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:02.172722 26021 raft_consensus.cc:504] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.174573 26021 leader_election.cc:288] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec434100daf64be49d5d76499a0f4080 (127.17.133.193:40507), b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821)
I20260925 14:53:02.183305 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.175759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:46957 (local address 127.17.133.194:36821)
0925 14:53:02.175939 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.175946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.177759 (+ 1813us) server_negotiation.cc:407] Connection header received
0925 14:53:02.177903 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.177907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.177963 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.178034 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:02.178607 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.179327 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.181281 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.182383 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.182562 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.182634 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:02.182903 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 14:53:02.183047 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:53:02.183476 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.175312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:54173 (local address 127.17.133.193:40507)
0925 14:53:02.177859 (+ 2547us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.177868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.177887 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:53:02.177963 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.177972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.178060 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.178161 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:02.178744 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.179479 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.181364 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.182465 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.182673 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.182743 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:02.183035 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 14:53:02.183171 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2433,"thread_start_us":2336,"threads_started":1}
I20260925 14:53:02.183539 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.175259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:36821 (local address 127.17.133.195:46957)
0925 14:53:02.177640 (+ 2381us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:02.177660 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:02.177797 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:02.178076 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:02.178083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:02.178114 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:02.178444 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.178454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.179455 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.179460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.181045 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.181059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.182548 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.182554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.182741 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.182764 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:53:02.182880 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 14:53:02.183004 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2259,"mutex_wait_us":47,"thread_start_us":2136,"threads_started":1}
I20260925 14:53:02.183832 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.174884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:40507 (local address 127.17.133.195:54173)
0925 14:53:02.175588 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:02.175608 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:02.175787 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:02.178207 (+ 2420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:02.178219 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:02.178252 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:02.178588 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.178599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.179609 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.179616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.181170 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.181186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.182701 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.182707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.182874 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.182897 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:53:02.183010 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:53:02.183153 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":18,"thread_start_us":574,"threads_started":1}
I20260925 14:53:02.184433 25802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9646f0fdd74f328b620f59a6e06445" candidate_uuid: "55ef238ea91343ff89e63c711f205c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1221b13e9374b78b54bb9c757e96241" is_pre_election: true
I20260925 14:53:02.184464 25667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9646f0fdd74f328b620f59a6e06445" candidate_uuid: "55ef238ea91343ff89e63c711f205c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec434100daf64be49d5d76499a0f4080" is_pre_election: true
I20260925 14:53:02.184710 25802 raft_consensus.cc:2454] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ef238ea91343ff89e63c711f205c22 in term 0.
I20260925 14:53:02.184711 25667 raft_consensus.cc:2454] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ef238ea91343ff89e63c711f205c22 in term 0.
I20260925 14:53:02.185227 25874 leader_election.cc:302] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55ef238ea91343ff89e63c711f205c22, b1221b13e9374b78b54bb9c757e96241, ec434100daf64be49d5d76499a0f4080; no voters:
I20260925 14:53:02.185600 26021 raft_consensus.cc:2781] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:02.185745 26021 raft_consensus.cc:482] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:02.185832 26021 raft_consensus.cc:3037] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:02.187868 26021 raft_consensus.cc:504] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.188418 26021 leader_election.cc:288] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [CANDIDATE]: Term 1 election: Requested vote from peers ec434100daf64be49d5d76499a0f4080 (127.17.133.193:40507), b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821)
I20260925 14:53:02.188915 25667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9646f0fdd74f328b620f59a6e06445" candidate_uuid: "55ef238ea91343ff89e63c711f205c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec434100daf64be49d5d76499a0f4080"
I20260925 14:53:02.189062 25802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9646f0fdd74f328b620f59a6e06445" candidate_uuid: "55ef238ea91343ff89e63c711f205c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1221b13e9374b78b54bb9c757e96241"
I20260925 14:53:02.189131 25667 raft_consensus.cc:3037] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:02.189282 25802 raft_consensus.cc:3037] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:02.191962 25802 raft_consensus.cc:2454] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ef238ea91343ff89e63c711f205c22 in term 1.
I20260925 14:53:02.191977 25667 raft_consensus.cc:2454] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ef238ea91343ff89e63c711f205c22 in term 1.
I20260925 14:53:02.192348 25873 leader_election.cc:302] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ef238ea91343ff89e63c711f205c22, b1221b13e9374b78b54bb9c757e96241; no voters:
I20260925 14:53:02.192710 26021 raft_consensus.cc:2781] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:02.193077 26021 raft_consensus.cc:686] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [term 1 LEADER]: Becoming Leader. State: Replica: 55ef238ea91343ff89e63c711f205c22, State: Running, Role: LEADER
I20260925 14:53:02.193487 26021 consensus_queue.cc:237] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.195185 26061 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: TxnStatusTablet state changed. Reason: New leader 55ef238ea91343ff89e63c711f205c22. Latest consensus state: current_term: 1 leader_uuid: "55ef238ea91343ff89e63c711f205c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:02.195559 26061 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:53:02.196235 26063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:53:02.196447 26063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:53:02.196942 25505 catalog_manager.cc:5495] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55ef238ea91343ff89e63c711f205c22 (127.17.133.195). New cstate: current_term: 1 leader_uuid: "55ef238ea91343ff89e63c711f205c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } health_report { overall_health: HEALTHY } } }
I20260925 14:53:02.259927 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.253876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.17.133.195:38797)
0925 14:53:02.254056 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.254062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.254178 (+ 116us) server_negotiation.cc:407] Connection header received
0925 14:53:02.254508 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.254573 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.254675 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.254813 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:02.255221 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.256153 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.257560 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.258078 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.258211 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.258680 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:02.259290 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:02.259437 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:02.259566 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 14:53:02.259670 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:53:02.260044 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.253863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:38797 (local address 127.0.0.1:40388)
0925 14:53:02.254082 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:02.254102 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:02.254281 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:02.254845 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:02.254850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:02.254879 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:02.255081 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.255096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.256300 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.256308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.257418 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.257430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.258233 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.258239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.258439 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.258489 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:02.259418 (+ 929us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:02.259440 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:53:02.259542 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:53:02.259720 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 14:53:02.262929 26020 consensus_queue.cc:1035] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:02.265295 25667 raft_consensus.cc:1261] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Refusing update from remote peer 55ef238ea91343ff89e63c711f205c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:02.265358 25801 raft_consensus.cc:1261] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Refusing update from remote peer 55ef238ea91343ff89e63c711f205c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:02.265959 26021 consensus_queue.cc:1035] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:02.266552 26021 consensus_queue.cc:1035] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:02.285594 26020 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: TxnStatusTablet state changed. Reason: New leader 55ef238ea91343ff89e63c711f205c22. Latest consensus state: current_term: 1 leader_uuid: "55ef238ea91343ff89e63c711f205c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:02.285872 26020 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:02.289111 26020 consensus_queue.cc:1035] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:02.302443 26025 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: TxnStatusTablet state changed. Reason: New leader 55ef238ea91343ff89e63c711f205c22. Latest consensus state: current_term: 1 leader_uuid: "55ef238ea91343ff89e63c711f205c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:02.302778 26025 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:02.306974 26073 mvcc.cc:205] Tried to move back new op lower bound from 7333265335350317056 to 7333265335067570176. Current Snapshot: MvccSnapshot[applied={T|T < 7333265335350317056}]
I20260925 14:53:02.307608 26025 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55ef238ea91343ff89e63c711f205c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:02.307837 26025 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P b1221b13e9374b78b54bb9c757e96241: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:02.312258 26021 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55ef238ea91343ff89e63c711f205c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:02.312377 26061 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55ef238ea91343ff89e63c711f205c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:02.312711 26061 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:53:02.313145 26021 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P 55ef238ea91343ff89e63c711f205c22: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:53:02.318152 26020 tablet_replica.cc:439] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55ef238ea91343ff89e63c711f205c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } } }
I20260925 14:53:02.318385 26020 tablet_replica.cc:442] T 6f9646f0fdd74f328b620f59a6e06445 P ec434100daf64be49d5d76499a0f4080: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:02.327181 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.320487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.17.133.193:40507)
0925 14:53:02.320664 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.320673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.321172 (+ 499us) server_negotiation.cc:407] Connection header received
0925 14:53:02.321375 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.321380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.321470 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.321628 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:02.322111 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.323226 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.325180 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.325581 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.325673 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.326113 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:02.326626 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:02.326733 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:02.326849 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 14:53:02.326926 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:53:02.342784 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.336790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:47315 (local address 127.17.133.195:38797)
0925 14:53:02.337023 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.337032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.337489 (+ 457us) server_negotiation.cc:407] Connection header received
0925 14:53:02.337650 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.337653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.337712 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.337794 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:02.338301 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.339046 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.340963 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.341486 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.341574 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.342030 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:02.342151 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:02.342232 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:02.342390 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:53:02.342482 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:53:02.343008 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.336750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:38797 (local address 127.17.133.193:47315)
0925 14:53:02.337389 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:02.337408 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:02.337542 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:02.337829 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:02.337833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:02.337845 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:02.338162 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.338172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.339163 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.339168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.340813 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.340826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.341627 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.341634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.341807 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.341867 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:02.342269 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:02.342278 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 14:53:02.342369 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:53:02.342532 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":337,"threads_started":1}
I20260925 14:53:02.379966 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.373597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:43869 (local address 127.17.133.193:40507)
0925 14:53:02.373753 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.373759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.374511 (+ 752us) server_negotiation.cc:407] Connection header received
0925 14:53:02.374716 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.374721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.374814 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.374901 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:02.375460 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.376169 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.378047 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.378656 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.378775 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.379296 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:02.379402 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:02.379495 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:02.379596 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 14:53:02.379663 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:53:02.380074 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.373701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:40507 (local address 127.17.133.195:43869)
0925 14:53:02.374408 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:02.374427 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:02.374590 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:02.374914 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:02.374925 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:02.374937 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:02.375316 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.375328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.376300 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.376307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.377929 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:02.377941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.378959 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.378963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:02.379085 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.379134 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:02.379496 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:02.379509 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:53:02.379581 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:53:02.379699 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":324,"threads_started":1}
I20260925 14:53:02.410521 25504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:02.410784 25504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:02.414111 25504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668: Sending DeleteTablet for 3 replicas of tablet 088b9f01c3d848e7b28761a1c2185c86
I20260925 14:53:02.414847 25642 tablet_service.cc:1514] Processing DeleteTablet for tablet 088b9f01c3d848e7b28761a1c2185c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:02 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:41040
I20260925 14:53:02.415495 25780 tablet_service.cc:1514] Processing DeleteTablet for tablet 088b9f01c3d848e7b28761a1c2185c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:02 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:53892
I20260925 14:53:02.416167 25920 tablet_service.cc:1514] Processing DeleteTablet for tablet 088b9f01c3d848e7b28761a1c2185c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:02 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:40372
I20260925 14:53:02.440284 26092 tablet_replica.cc:330] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: stopping tablet replica
I20260925 14:53:02.440500 26092 raft_consensus.cc:2229] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:02.440786 26092 raft_consensus.cc:2258] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:02.441411 26091 tablet_replica.cc:330] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: stopping tablet replica
I20260925 14:53:02.441613 26091 raft_consensus.cc:2229] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:02.441847 26091 raft_consensus.cc:2258] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:02.443091 26092 ts_tablet_manager.cc:1905] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:02.444195 26091 ts_tablet_manager.cc:1905] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:02.448185 26090 tablet_replica.cc:330] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: stopping tablet replica
I20260925 14:53:02.454715 26092 ts_tablet_manager.cc:1918] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:02.454726 26091 ts_tablet_manager.cc:1918] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:02.454859 26092 log.cc:1196] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/088b9f01c3d848e7b28761a1c2185c86
I20260925 14:53:02.454881 26091 log.cc:1196] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/088b9f01c3d848e7b28761a1c2185c86
I20260925 14:53:02.455365 26092 ts_tablet_manager.cc:1939] T 088b9f01c3d848e7b28761a1c2185c86 P 55ef238ea91343ff89e63c711f205c22: Deleting consensus metadata
I20260925 14:53:02.455363 26091 ts_tablet_manager.cc:1939] T 088b9f01c3d848e7b28761a1c2185c86 P b1221b13e9374b78b54bb9c757e96241: Deleting consensus metadata
I20260925 14:53:02.448390 26090 raft_consensus.cc:2229] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:02.456656 25491 catalog_manager.cc:4841] TS b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821): tablet 088b9f01c3d848e7b28761a1c2185c86 (table test-table [id=67a1570734984e85826f44f06fa99b86]) successfully deleted
I20260925 14:53:02.456925 25491 catalog_manager.cc:4841] TS 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797): tablet 088b9f01c3d848e7b28761a1c2185c86 (table test-table [id=67a1570734984e85826f44f06fa99b86]) successfully deleted
I20260925 14:53:02.456947 26090 raft_consensus.cc:2258] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:02.461347 26090 ts_tablet_manager.cc:1905] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:02.469111 26090 ts_tablet_manager.cc:1918] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:02.469270 26090 log.cc:1196] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/088b9f01c3d848e7b28761a1c2185c86
I20260925 14:53:02.469743 26090 ts_tablet_manager.cc:1939] T 088b9f01c3d848e7b28761a1c2185c86 P ec434100daf64be49d5d76499a0f4080: Deleting consensus metadata
I20260925 14:53:02.470890 25492 catalog_manager.cc:4841] TS ec434100daf64be49d5d76499a0f4080 (127.17.133.193:40507): tablet 088b9f01c3d848e7b28761a1c2185c86 (table test-table [id=67a1570734984e85826f44f06fa99b86]) successfully deleted
Sep 25 14:53:02 dist-test-slave-v5vl krb5kdc[25460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:53:02 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:02 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:53:02.522367 26100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:02.490537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.17.133.254:33163)
0925 14:53:02.505810 (+ 15273us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.505817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.508994 (+ 3177us) server_negotiation.cc:407] Connection header received
0925 14:53:02.509061 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.509067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.509150 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.509243 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:02.511145 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.512459 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.514205 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.514908 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.514996 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.522079 (+ 7083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54714: BlockingRecv error: recv got EOF from 127.0.0.1:54714 (error 108)
Metrics: {"server-negotiator.queue_time_us":15116,"thread_start_us":15283,"threads_started":1}
Sep 25 14:53:02 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM
I20260925 14:53:02.561944 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.543675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.17.133.254:33163)
0925 14:53:02.543876 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.543882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.544372 (+ 490us) server_negotiation.cc:407] Connection header received
0925 14:53:02.544609 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.544615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.544698 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.544822 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:02.546409 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.547562 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.549493 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.550125 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.550269 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.555268 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:02.555275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:02.555281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:02.555322 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:02.558986 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:02.559775 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:02.559781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:02.559785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:02.559924 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:02.560403 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:02.560409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:02.560414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:02.560808 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:02.561045 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:02.561365 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 14:53:02.561588 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:53:02.566272 25504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:02.567001 25504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:02.574906 25642 tablet_service.cc:1467] Processing CreateTablet for tablet fd6da305f26149e3bfda4e72cca1be51 (DEFAULT_TABLE table=test-table [id=711124c2166b4e72a1e6c87d408201ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:02.575348 25642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6da305f26149e3bfda4e72cca1be51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:02.575270 25920 tablet_service.cc:1467] Processing CreateTablet for tablet fd6da305f26149e3bfda4e72cca1be51 (DEFAULT_TABLE table=test-table [id=711124c2166b4e72a1e6c87d408201ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:02.575713 25920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6da305f26149e3bfda4e72cca1be51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:02.576046 25780 tablet_service.cc:1467] Processing CreateTablet for tablet fd6da305f26149e3bfda4e72cca1be51 (DEFAULT_TABLE table=test-table [id=711124c2166b4e72a1e6c87d408201ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:02.576490 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6da305f26149e3bfda4e72cca1be51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:02.589228 26108 tablet_bootstrap.cc:492] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: Bootstrap starting.
I20260925 14:53:02.590999 26108 tablet_bootstrap.cc:654] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:02.592674 26106 tablet_bootstrap.cc:492] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: Bootstrap starting.
I20260925 14:53:02.593259 26107 tablet_bootstrap.cc:492] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: Bootstrap starting.
I20260925 14:53:02.595760 26107 tablet_bootstrap.cc:654] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:02.595942 26106 tablet_bootstrap.cc:654] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:02.598579 26108 tablet_bootstrap.cc:492] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: No bootstrap required, opened a new log
I20260925 14:53:02.598712 26108 ts_tablet_manager.cc:1397] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260925 14:53:02.599459 26108 raft_consensus.cc:348] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.599706 26108 raft_consensus.cc:374] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:02.599792 26108 raft_consensus.cc:729] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1221b13e9374b78b54bb9c757e96241, State: Initialized, Role: FOLLOWER
I20260925 14:53:02.600029 26108 consensus_queue.cc:260] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.601230 26108 ts_tablet_manager.cc:1428] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:02.606156 26107 tablet_bootstrap.cc:492] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: No bootstrap required, opened a new log
I20260925 14:53:02.606200 26106 tablet_bootstrap.cc:492] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: No bootstrap required, opened a new log
I20260925 14:53:02.606338 26106 ts_tablet_manager.cc:1397] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20260925 14:53:02.606364 26107 ts_tablet_manager.cc:1397] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20260925 14:53:02.607053 26106 raft_consensus.cc:348] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.607293 26106 raft_consensus.cc:374] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:02.607400 26106 raft_consensus.cc:729] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec434100daf64be49d5d76499a0f4080, State: Initialized, Role: FOLLOWER
I20260925 14:53:02.607177 26107 raft_consensus.cc:348] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.607636 26107 raft_consensus.cc:374] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:02.607651 26106 consensus_queue.cc:260] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.607788 26107 raft_consensus.cc:729] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ef238ea91343ff89e63c711f205c22, State: Initialized, Role: FOLLOWER
I20260925 14:53:02.608155 26107 consensus_queue.cc:260] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.608794 26107 ts_tablet_manager.cc:1428] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 14:53:02.611800 26106 ts_tablet_manager.cc:1428] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260925 14:53:02.704123 26072 raft_consensus.cc:482] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:02.704351 26072 raft_consensus.cc:504] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.705158 26072 leader_election.cc:288] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821), 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797)
I20260925 14:53:02.705610 25802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6da305f26149e3bfda4e72cca1be51" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1221b13e9374b78b54bb9c757e96241" is_pre_election: true
I20260925 14:53:02.705739 25940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6da305f26149e3bfda4e72cca1be51" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ef238ea91343ff89e63c711f205c22" is_pre_election: true
I20260925 14:53:02.705847 25802 raft_consensus.cc:2454] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 0.
I20260925 14:53:02.706024 25940 raft_consensus.cc:2454] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 0.
I20260925 14:53:02.706209 25594 leader_election.cc:302] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1221b13e9374b78b54bb9c757e96241, ec434100daf64be49d5d76499a0f4080; no voters:
I20260925 14:53:02.706485 26072 raft_consensus.cc:2781] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:02.706660 26072 raft_consensus.cc:482] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:02.706753 26072 raft_consensus.cc:3037] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:02.709986 26072 raft_consensus.cc:504] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.710580 26072 leader_election.cc:288] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 election: Requested vote from peers b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821), 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797)
I20260925 14:53:02.711056 25802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6da305f26149e3bfda4e72cca1be51" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1221b13e9374b78b54bb9c757e96241"
I20260925 14:53:02.711249 25802 raft_consensus.cc:3037] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:02.711179 25940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6da305f26149e3bfda4e72cca1be51" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ef238ea91343ff89e63c711f205c22"
I20260925 14:53:02.711391 25940 raft_consensus.cc:3037] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:02.714052 25802 raft_consensus.cc:2454] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 1.
I20260925 14:53:02.714048 25940 raft_consensus.cc:2454] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 1.
I20260925 14:53:02.714447 25594 leader_election.cc:302] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1221b13e9374b78b54bb9c757e96241, ec434100daf64be49d5d76499a0f4080; no voters:
I20260925 14:53:02.714704 26072 raft_consensus.cc:2781] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:02.714960 26072 raft_consensus.cc:686] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 1 LEADER]: Becoming Leader. State: Replica: ec434100daf64be49d5d76499a0f4080, State: Running, Role: LEADER
I20260925 14:53:02.715278 26072 consensus_queue.cc:237] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.718206 25504 catalog_manager.cc:5495] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec434100daf64be49d5d76499a0f4080 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "ec434100daf64be49d5d76499a0f4080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:02.744716 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.739036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.17.133.193:40507)
0925 14:53:02.739232 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.739238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.739358 (+ 120us) server_negotiation.cc:407] Connection header received
0925 14:53:02.739524 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.739538 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.739637 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.739747 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:02.740050 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.741223 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.742596 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.743327 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.743493 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.743951 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:02.744087 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:02.744182 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:02.744294 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 14:53:02.744377 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:53:02.747283 25940 raft_consensus.cc:1261] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Refusing update from remote peer ec434100daf64be49d5d76499a0f4080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:02.747371 25802 raft_consensus.cc:1261] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Refusing update from remote peer ec434100daf64be49d5d76499a0f4080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:02.747934 26072 consensus_queue.cc:1035] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:02.748190 26046 consensus_queue.cc:1035] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:02.759032 26066 mvcc.cc:205] Tried to move back new op lower bound from 7333265337327284224 to 7333265337204297728. Current Snapshot: MvccSnapshot[applied={T|T < 7333265337327284224 or (T in {7333265337327284224})}]
I20260925 14:53:02.762516 25505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:02.762758 25505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:02.767630 25780 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6da305f26149e3bfda4e72cca1be51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:02 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:53892
I20260925 14:53:02.767941 25505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668: Sending DeleteTablet for 3 replicas of tablet fd6da305f26149e3bfda4e72cca1be51
I20260925 14:53:02.768765 25780 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6da305f26149e3bfda4e72cca1be51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:02 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:53892
I20260925 14:53:02.768805 25642 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6da305f26149e3bfda4e72cca1be51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:02 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:41040
I20260925 14:53:02.768880 26091 tablet_replica.cc:330] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: stopping tablet replica
I20260925 14:53:02.769050 26090 tablet_replica.cc:330] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: stopping tablet replica
I20260925 14:53:02.769060 26091 raft_consensus.cc:2229] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:02.769258 26091 raft_consensus.cc:2258] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:02.769265 26090 raft_consensus.cc:2229] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:02.769557 26090 raft_consensus.cc:2258] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:02.770571 26091 ts_tablet_manager.cc:1905] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:02.770589 26090 ts_tablet_manager.cc:1905] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:02.772764 25920 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6da305f26149e3bfda4e72cca1be51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:02 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:40372
I20260925 14:53:02.773039 26092 tablet_replica.cc:330] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: stopping tablet replica
I20260925 14:53:02.773236 26092 raft_consensus.cc:2229] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:02.773444 26092 raft_consensus.cc:2258] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:02.775182 26091 ts_tablet_manager.cc:1918] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:02.775318 26091 log.cc:1196] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/fd6da305f26149e3bfda4e72cca1be51
I20260925 14:53:02.775785 26091 ts_tablet_manager.cc:1939] T fd6da305f26149e3bfda4e72cca1be51 P b1221b13e9374b78b54bb9c757e96241: Deleting consensus metadata
I20260925 14:53:02.776609 26092 ts_tablet_manager.cc:1905] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:02.776798 25491 catalog_manager.cc:4841] TS b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821): tablet fd6da305f26149e3bfda4e72cca1be51 (table test-table [id=711124c2166b4e72a1e6c87d408201ba]) successfully deleted
W20260925 14:53:02.777642 25491 catalog_manager.cc:4808] TS b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821): delete failed for tablet fd6da305f26149e3bfda4e72cca1be51 because the tablet was not found. No further retry: Not found: Tablet not found: fd6da305f26149e3bfda4e72cca1be51
I20260925 14:53:02.778861 25780 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6da305f26149e3bfda4e72cca1be51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:02 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:53892
W20260925 14:53:02.779331 25491 catalog_manager.cc:4808] TS b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821): delete failed for tablet fd6da305f26149e3bfda4e72cca1be51 because the tablet was not found. No further retry: Not found: Tablet not found: fd6da305f26149e3bfda4e72cca1be51
I20260925 14:53:02.781621 26092 ts_tablet_manager.cc:1918] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:02.781780 26092 log.cc:1196] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/fd6da305f26149e3bfda4e72cca1be51
I20260925 14:53:02.782153 26092 ts_tablet_manager.cc:1939] T fd6da305f26149e3bfda4e72cca1be51 P 55ef238ea91343ff89e63c711f205c22: Deleting consensus metadata
I20260925 14:53:02.782264 26090 ts_tablet_manager.cc:1918] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:02.782426 26090 log.cc:1196] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/fd6da305f26149e3bfda4e72cca1be51
I20260925 14:53:02.782886 26090 ts_tablet_manager.cc:1939] T fd6da305f26149e3bfda4e72cca1be51 P ec434100daf64be49d5d76499a0f4080: Deleting consensus metadata
I20260925 14:53:02.783178 25491 catalog_manager.cc:4841] TS 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797): tablet fd6da305f26149e3bfda4e72cca1be51 (table test-table [id=711124c2166b4e72a1e6c87d408201ba]) successfully deleted
I20260925 14:53:02.783273 25505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:02.783883 25505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:02.784448 25492 catalog_manager.cc:4841] TS ec434100daf64be49d5d76499a0f4080 (127.17.133.193:40507): tablet fd6da305f26149e3bfda4e72cca1be51 (table test-table [id=711124c2166b4e72a1e6c87d408201ba]) successfully deleted
I20260925 14:53:02.790759 25780 tablet_service.cc:1467] Processing CreateTablet for tablet 25b5ddf7f9fc42faa0e30a5f70e4c687 (DEFAULT_TABLE table=test-table [id=5bb966dd276a4f5380b6965173aafae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:02.790792 25642 tablet_service.cc:1467] Processing CreateTablet for tablet 25b5ddf7f9fc42faa0e30a5f70e4c687 (DEFAULT_TABLE table=test-table [id=5bb966dd276a4f5380b6965173aafae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:02.791164 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b5ddf7f9fc42faa0e30a5f70e4c687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:02.791195 25642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b5ddf7f9fc42faa0e30a5f70e4c687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:02.797524 26108 tablet_bootstrap.cc:492] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: Bootstrap starting.
I20260925 14:53:02.797513 25920 tablet_service.cc:1467] Processing CreateTablet for tablet 25b5ddf7f9fc42faa0e30a5f70e4c687 (DEFAULT_TABLE table=test-table [id=5bb966dd276a4f5380b6965173aafae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:02.797942 25920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b5ddf7f9fc42faa0e30a5f70e4c687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:02.799096 26108 tablet_bootstrap.cc:654] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:02.804418 26106 tablet_bootstrap.cc:492] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: Bootstrap starting.
I20260925 14:53:02.804811 26107 tablet_bootstrap.cc:492] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: Bootstrap starting.
I20260925 14:53:02.805001 26108 tablet_bootstrap.cc:492] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: No bootstrap required, opened a new log
I20260925 14:53:02.805209 26108 ts_tablet_manager.cc:1397] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260925 14:53:02.806010 26108 raft_consensus.cc:348] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.806353 26107 tablet_bootstrap.cc:654] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:02.806334 26108 raft_consensus.cc:374] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:02.806489 26108 raft_consensus.cc:729] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1221b13e9374b78b54bb9c757e96241, State: Initialized, Role: FOLLOWER
I20260925 14:53:02.806816 26108 consensus_queue.cc:260] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.807217 26106 tablet_bootstrap.cc:654] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:02.807617 26108 ts_tablet_manager.cc:1428] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:02.809525 26107 tablet_bootstrap.cc:492] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: No bootstrap required, opened a new log
I20260925 14:53:02.809738 26107 ts_tablet_manager.cc:1397] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 14:53:02.810766 26107 raft_consensus.cc:348] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.811098 26107 raft_consensus.cc:374] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:02.811201 26107 raft_consensus.cc:729] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ef238ea91343ff89e63c711f205c22, State: Initialized, Role: FOLLOWER
I20260925 14:53:02.811532 26107 consensus_queue.cc:260] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.813004 26107 ts_tablet_manager.cc:1428] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:02.814973 26106 tablet_bootstrap.cc:492] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: No bootstrap required, opened a new log
I20260925 14:53:02.815166 26106 ts_tablet_manager.cc:1397] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 14:53:02.816004 26106 raft_consensus.cc:348] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.816322 26106 raft_consensus.cc:374] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:02.816421 26106 raft_consensus.cc:729] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec434100daf64be49d5d76499a0f4080, State: Initialized, Role: FOLLOWER
I20260925 14:53:02.816740 26106 consensus_queue.cc:260] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.817234 26106 ts_tablet_manager.cc:1428] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:02.929852 26072 raft_consensus.cc:482] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:02.930022 26072 raft_consensus.cc:504] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.930586 26072 leader_election.cc:288] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821), 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797)
I20260925 14:53:02.931010 25802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b5ddf7f9fc42faa0e30a5f70e4c687" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1221b13e9374b78b54bb9c757e96241" is_pre_election: true
I20260925 14:53:02.931092 25940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b5ddf7f9fc42faa0e30a5f70e4c687" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ef238ea91343ff89e63c711f205c22" is_pre_election: true
I20260925 14:53:02.931222 25802 raft_consensus.cc:2454] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 0.
I20260925 14:53:02.931398 25940 raft_consensus.cc:2454] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 0.
I20260925 14:53:02.931685 25594 leader_election.cc:302] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1221b13e9374b78b54bb9c757e96241, ec434100daf64be49d5d76499a0f4080; no voters:
I20260925 14:53:02.931903 26072 raft_consensus.cc:2781] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:02.931998 26072 raft_consensus.cc:482] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:02.932117 26072 raft_consensus.cc:3037] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:02.934094 26072 raft_consensus.cc:504] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.934538 26072 leader_election.cc:288] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 election: Requested vote from peers b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821), 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797)
I20260925 14:53:02.934934 25802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b5ddf7f9fc42faa0e30a5f70e4c687" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1221b13e9374b78b54bb9c757e96241"
I20260925 14:53:02.935034 25940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b5ddf7f9fc42faa0e30a5f70e4c687" candidate_uuid: "ec434100daf64be49d5d76499a0f4080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ef238ea91343ff89e63c711f205c22"
I20260925 14:53:02.935123 25802 raft_consensus.cc:3037] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:02.935256 25940 raft_consensus.cc:3037] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:02.937865 25940 raft_consensus.cc:2454] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 1.
I20260925 14:53:02.937883 25802 raft_consensus.cc:2454] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec434100daf64be49d5d76499a0f4080 in term 1.
I20260925 14:53:02.938169 25594 leader_election.cc:302] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1221b13e9374b78b54bb9c757e96241, ec434100daf64be49d5d76499a0f4080; no voters:
I20260925 14:53:02.938407 26072 raft_consensus.cc:2781] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:02.938566 26072 raft_consensus.cc:686] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 1 LEADER]: Becoming Leader. State: Replica: ec434100daf64be49d5d76499a0f4080, State: Running, Role: LEADER
I20260925 14:53:02.938823 26072 consensus_queue.cc:237] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } }
I20260925 14:53:02.941080 25505 catalog_manager.cc:5495] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec434100daf64be49d5d76499a0f4080 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "ec434100daf64be49d5d76499a0f4080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec434100daf64be49d5d76499a0f4080" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:02.966558 25940 raft_consensus.cc:1261] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Refusing update from remote peer ec434100daf64be49d5d76499a0f4080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:02.966583 25802 raft_consensus.cc:1261] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Refusing update from remote peer ec434100daf64be49d5d76499a0f4080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:02.967046 26072 consensus_queue.cc:1035] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1221b13e9374b78b54bb9c757e96241" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 36821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:02.967324 26046 consensus_queue.cc:1035] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ef238ea91343ff89e63c711f205c22" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:03.006394 25505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:03.006652 25505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:03.009230 25505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4cb4d48255f4972aaa107c3da5c0668: Sending DeleteTablet for 3 replicas of tablet 25b5ddf7f9fc42faa0e30a5f70e4c687
I20260925 14:53:03.009835 25780 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b5ddf7f9fc42faa0e30a5f70e4c687 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:03 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:53892
I20260925 14:53:03.009915 25642 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b5ddf7f9fc42faa0e30a5f70e4c687 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:03 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:41040
I20260925 14:53:03.010016 25920 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b5ddf7f9fc42faa0e30a5f70e4c687 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:03 UTC) from {username='oryx', principal='oryx/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:40372
I20260925 14:53:03.010109 26091 tablet_replica.cc:330] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: stopping tablet replica
I20260925 14:53:03.010177 26090 tablet_replica.cc:330] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: stopping tablet replica
I20260925 14:53:03.010277 26092 tablet_replica.cc:330] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: stopping tablet replica
I20260925 14:53:03.010315 26091 raft_consensus.cc:2229] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:03.010340 26090 raft_consensus.cc:2229] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:03.010432 26092 raft_consensus.cc:2229] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:03.010501 26091 raft_consensus.cc:2258] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:03.010596 26092 raft_consensus.cc:2258] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:03.010628 26090 raft_consensus.cc:2258] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:03.011034 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 25576
I20260925 14:53:03.011567 26091 ts_tablet_manager.cc:1905] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:03.011714 26092 ts_tablet_manager.cc:1905] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:03.012871 26090 ts_tablet_manager.cc:1905] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:03.016330 26091 ts_tablet_manager.cc:1918] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:03.016467 26091 log.cc:1196] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/25b5ddf7f9fc42faa0e30a5f70e4c687
I20260925 14:53:03.016949 26091 ts_tablet_manager.cc:1939] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P b1221b13e9374b78b54bb9c757e96241: Deleting consensus metadata
I20260925 14:53:03.018024 26092 ts_tablet_manager.cc:1918] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:03.018092 25491 catalog_manager.cc:4841] TS b1221b13e9374b78b54bb9c757e96241 (127.17.133.194:36821): tablet 25b5ddf7f9fc42faa0e30a5f70e4c687 (table test-table [id=5bb966dd276a4f5380b6965173aafae0]) successfully deleted
I20260925 14:53:03.018154 26092 log.cc:1196] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/25b5ddf7f9fc42faa0e30a5f70e4c687
I20260925 14:53:03.018553 26092 ts_tablet_manager.cc:1939] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P 55ef238ea91343ff89e63c711f205c22: Deleting consensus metadata
I20260925 14:53:03.019215 26090 ts_tablet_manager.cc:1918] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:03.019342 26090 log.cc:1196] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/25b5ddf7f9fc42faa0e30a5f70e4c687
I20260925 14:53:03.019518 25491 catalog_manager.cc:4841] TS 55ef238ea91343ff89e63c711f205c22 (127.17.133.195:38797): tablet 25b5ddf7f9fc42faa0e30a5f70e4c687 (table test-table [id=5bb966dd276a4f5380b6965173aafae0]) successfully deleted
I20260925 14:53:03.019745 26090 ts_tablet_manager.cc:1939] T 25b5ddf7f9fc42faa0e30a5f70e4c687 P ec434100daf64be49d5d76499a0f4080: Deleting consensus metadata
I20260925 14:53:03.020496 25492 catalog_manager.cc:4841] TS ec434100daf64be49d5d76499a0f4080 (127.17.133.193:40507): tablet 25b5ddf7f9fc42faa0e30a5f70e4c687 (table test-table [id=5bb966dd276a4f5380b6965173aafae0]) successfully deleted
Sep 25 14:53:03 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.193@KRBTEST.COM
I20260925 14:53:03.033000 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:03.017185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.17.133.193:40507)
0925 14:53:03.017339 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:03.017345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:03.020820 (+ 3475us) server_negotiation.cc:407] Connection header received
0925 14:53:03.020934 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:03.020938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:03.021015 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:03.021097 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:03.022201 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.022856 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.024084 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.024558 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.024696 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:03.028340 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:03.028353 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:03.028368 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:03.028394 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:03.030804 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:03.031383 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:03.031390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:03.031392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:03.031500 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:03.031783 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:03.031796 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:03.031798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:03.032100 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:03.032322 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:03.032555 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 14:53:03.032763 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:53:03.033577 25707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:03.154147 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 25576
I20260925 14:53:03.182953 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 25716
Sep 25 14:53:03 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.194@KRBTEST.COM
I20260925 14:53:03.197867 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:03.183755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.17.133.194:36821)
0925 14:53:03.184234 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:03.184238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:03.184252 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:53:03.184299 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:03.184302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:03.184381 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:03.184493 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:03.185780 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.186589 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.187786 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.188235 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.188366 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:03.192561 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:03.192573 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:03.192646 (+ 73us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:03.192677 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:03.195319 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:03.196028 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:03.196034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:03.196037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:03.196139 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:03.196481 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:03.196484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:03.196486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:03.196833 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:03.197079 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:03.197346 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 14:53:03.197494 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260925 14:53:03.198503 25846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:03.317653 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 25716
I20260925 14:53:03.339507 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 25855
Sep 25 14:53:03 dist-test-slave-v5vl krb5kdc[25460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.133.195@KRBTEST.COM
I20260925 14:53:03.353727 26131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:03.340171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.17.133.195:38797)
0925 14:53:03.340569 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:03.340573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:03.340639 (+ 66us) server_negotiation.cc:407] Connection header received
0925 14:53:03.340691 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:03.340695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:03.340751 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:03.340835 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:03.342057 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.342743 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.343915 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.344287 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.344371 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:03.348488 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:03.348508 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:03.348530 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:03.348580 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:03.351154 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:03.351725 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:03.351731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:03.351734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:03.351851 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:03.352134 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:03.352137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:03.352139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:03.352447 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:03.352746 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:03.353202 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 14:53:03.353420 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":209,"threads_started":1}
I20260925 14:53:03.354329 25984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:03.471895 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 25855
I20260925 14:53:03.493052 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 25473
I20260925 14:53:03.494100 25545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:03.606325 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 25473
2026-09-25T14:53:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5798 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:03 dist-test-slave-v5vl krb5kdc[26137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:03 dist-test-slave-v5vl krb5kdc[26137](info): set up 2 sockets
Sep 25 14:53:03 dist-test-slave-v5vl krb5kdc[26137](info): commencing operation
krb5kdc: starting...
W20260925 14:53:04.689460 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:53:04 dist-test-slave-v5vl krb5kdc[26137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:53:04.890139 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37151
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:37151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:05.199060 26150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:05.199337 26150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:05.199525 26150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:05.206682 26150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:05.206780 26150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:05.206832 26150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:05.206951 26150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:05.207041 26150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:05.216415 26150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:37151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:05.217648 26150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:05.219048 26150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:05.230655 26158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.230676 26156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.230710 26155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.231858 26150 server_base.cc:1034] running on GCE node
I20260925 14:53:05.232671 26150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:05.233940 26150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:05.235136 26150 hybrid_clock.cc:648] HybridClock initialized: now 1790347985235071 us; error 52 us; skew 500 ppm
I20260925 14:53:05.235481 26150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T14:53:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1697 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:05 dist-test-slave-v5vl krb5kdc[26164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:05 dist-test-slave-v5vl krb5kdc[26164](info): set up 2 sockets
Sep 25 14:53:05 dist-test-slave-v5vl krb5kdc[26164](info): commencing operation
krb5kdc: starting...
W20260925 14:53:06.382170 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:53:06 dist-test-slave-v5vl krb5kdc[26164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:53:06.592144 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45591
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:06.905033 26177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:06.905323 26177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:06.905488 26177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:06.912667 26177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:06.912770 26177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:06.912889 26177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:06.912992 26177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:06.913075 26177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:06.923054 26177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:45591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:06.924501 26177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:06.925897 26177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:06.937428 26182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:06.937378 26183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:06.937342 26185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:06.938301 26177 server_base.cc:1034] running on GCE node
I20260925 14:53:06.939026 26177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:06.940368 26177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:06.941557 26177 hybrid_clock.cc:648] HybridClock initialized: now 1790347986941517 us; error 32 us; skew 500 ppm
I20260925 14:53:06.941893 26177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:06 dist-test-slave-v5vl krb5kdc[26164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347986, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:06.944855 26177 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:53:06.946166 26177 webserver.cc:460] Webserver started at http://127.17.133.254:33291/ using document root <none> and password file <none>
I20260925 14:53:06.946756 26177 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:06.946851 26177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:06.947094 26177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:06.949869 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "a4483e17140f4b719b0159f7484875b4"
format_stamp: "Formatted at 2026-09-25 14:53:06 on dist-test-slave-v5vl"
server_key: "49b97de04525b0f89365a67fc9ddaec5"
server_key_iv: "6a7f706c41d5203357ddeed001a46d43"
server_key_version: "encryptionkey@0"
I20260925 14:53:06.950517 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "a4483e17140f4b719b0159f7484875b4"
format_stamp: "Formatted at 2026-09-25 14:53:06 on dist-test-slave-v5vl"
server_key: "49b97de04525b0f89365a67fc9ddaec5"
server_key_iv: "6a7f706c41d5203357ddeed001a46d43"
server_key_version: "encryptionkey@0"
I20260925 14:53:06.955384 26177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 14:53:06.958419 26192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:06.959592 26177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 14:53:06.959765 26177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "a4483e17140f4b719b0159f7484875b4"
format_stamp: "Formatted at 2026-09-25 14:53:06 on dist-test-slave-v5vl"
server_key: "49b97de04525b0f89365a67fc9ddaec5"
server_key_iv: "6a7f706c41d5203357ddeed001a46d43"
server_key_version: "encryptionkey@0"
I20260925 14:53:06.959937 26177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:06.982713 26177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:06.986379 26177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:06.986655 26177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:07.004082 26177 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:45591
I20260925 14:53:07.004101 26243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:45591 every 8 connection(s)
I20260925 14:53:07.005568 26177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:07.009296 26244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:07.010890 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 26177
I20260925 14:53:07.011106 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:53:07.011714 17943 external_mini_cluster.cc:1428] Setting key 639357ca6f0f9ad2b94f8c55e3f784ef
I20260925 14:53:07.022202 26244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: Bootstrap starting.
I20260925 14:53:07.025734 26244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: Neither blocks nor log segments found. Creating new log.
Sep 25 14:53:07 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:53:07.026715 26244 log.cc:824] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:07.029672 26244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: No bootstrap required, opened a new log
I20260925 14:53:07.034144 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.013737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.17.133.254:45591)
0925 14:53:07.014554 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:07.014569 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:07.014620 (+ 51us) server_negotiation.cc:407] Connection header received
0925 14:53:07.016991 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:07.017020 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:07.017506 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:07.017916 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:07.019140 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.021224 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.022446 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.023448 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.023621 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:07.027362 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:07.027370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:07.027383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:07.027421 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:07.030438 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:07.031068 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:07.031077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:07.031082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:07.031247 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:07.031620 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:07.031627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:07.031630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:07.032105 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:07.032420 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:07.032887 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 14:53:07.033238 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":342,"threads_started":1}
I20260925 14:53:07.037276 26244 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } }
I20260925 14:53:07.037637 26244 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:07.037746 26244 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4483e17140f4b719b0159f7484875b4, State: Initialized, Role: FOLLOWER
I20260925 14:53:07.038314 26244 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } }
I20260925 14:53:07.038534 26244 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:07.038663 26244 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:07.038837 26244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:07.041246 26244 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } }
I20260925 14:53:07.041698 26244 leader_election.cc:302] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4483e17140f4b719b0159f7484875b4; no voters:
I20260925 14:53:07.042239 26244 leader_election.cc:288] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:07.042357 26249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:07.043074 26249 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 1 LEADER]: Becoming Leader. State: Replica: a4483e17140f4b719b0159f7484875b4, State: Running, Role: LEADER
I20260925 14:53:07.043582 26249 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } }
I20260925 14:53:07.044294 26244 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:07.046072 26251 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4483e17140f4b719b0159f7484875b4. Latest consensus state: current_term: 1 leader_uuid: "a4483e17140f4b719b0159f7484875b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } } }
I20260925 14:53:07.046583 26251 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:07.046303 26250 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4483e17140f4b719b0159f7484875b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } } }
I20260925 14:53:07.046733 26250 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:07.047399 26254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:07.052356 26254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:07.061735 26254 catalog_manager.cc:1347] Generated new cluster ID: f286c41257eb467faf56c38f476e24d7
I20260925 14:53:07.061860 26254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:07.075174 26254 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:07.076342 26254 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:07.081526 26254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: Generated new TSK 0
I20260925 14:53:07.082263 26254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:53:07.143728 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45591
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:07.477340 26270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:07.477600 26270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:07.477690 26270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:07.484541 26270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:07.484704 26270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:07.484830 26270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:53:07.494292 26270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:07.495676 26270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:07.497066 26270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:07.510062 26275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:07.510150 26276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:07.510757 26278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:07.510831 26270 server_base.cc:1034] running on GCE node
I20260925 14:53:07.511545 26270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:07.512521 26270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:07.513806 26270 hybrid_clock.cc:648] HybridClock initialized: now 1790347987513754 us; error 39 us; skew 500 ppm
I20260925 14:53:07.514149 26270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:07 dist-test-slave-v5vl krb5kdc[26164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347987, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:07.518090 26270 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:53:07.519536 26270 webserver.cc:460] Webserver started at http://127.17.133.193:35585/ using document root <none> and password file <none>
I20260925 14:53:07.520243 26270 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:07.520377 26270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:07.520761 26270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:07.523758 26270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "d2d554448c3c405c87887c42a251447e"
format_stamp: "Formatted at 2026-09-25 14:53:07 on dist-test-slave-v5vl"
server_key: "12476bcbce1c4b87b6e81c2cc874e58d"
server_key_iv: "a841e2d6a49760fe0ce500cee4bf9c80"
server_key_version: "encryptionkey@0"
I20260925 14:53:07.524539 26270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "d2d554448c3c405c87887c42a251447e"
format_stamp: "Formatted at 2026-09-25 14:53:07 on dist-test-slave-v5vl"
server_key: "12476bcbce1c4b87b6e81c2cc874e58d"
server_key_iv: "a841e2d6a49760fe0ce500cee4bf9c80"
server_key_version: "encryptionkey@0"
I20260925 14:53:07.529563 26270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 14:53:07.532903 26285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:07.534442 26270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:53:07.534600 26270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "d2d554448c3c405c87887c42a251447e"
format_stamp: "Formatted at 2026-09-25 14:53:07 on dist-test-slave-v5vl"
server_key: "12476bcbce1c4b87b6e81c2cc874e58d"
server_key_iv: "a841e2d6a49760fe0ce500cee4bf9c80"
server_key_version: "encryptionkey@0"
I20260925 14:53:07.534790 26270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:07.549050 26270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:07.552750 26270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:07.553143 26270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:07.554108 26270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:07.555351 26270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:07.555418 26270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:07.555552 26270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:07.555648 26270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:07.588384 26270 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:37921
I20260925 14:53:07.588430 26397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:37921 every 8 connection(s)
I20260925 14:53:07.589655 26270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:07.597839 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 26270
I20260925 14:53:07.598047 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:07.598651 17943 external_mini_cluster.cc:1428] Setting key 386d41e1e43661ad9cc23606e25ecfa7
Sep 25 14:53:07 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347987, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:53:07.610229 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.592050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:52539 (local address 127.17.133.254:45591)
0925 14:53:07.592472 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:07.592477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:07.593373 (+ 896us) server_negotiation.cc:407] Connection header received
0925 14:53:07.595137 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:07.595143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:07.595235 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:07.595353 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:07.596823 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.597592 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.599349 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.599869 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.599969 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:07.603581 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:07.603589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:07.603596 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:07.603640 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:07.607086 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:07.607765 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:07.607772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:07.607777 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:07.607923 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:07.608324 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:07.608331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:07.608337 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:07.608738 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:07.608885 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:07.609524 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 14:53:07.609759 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":228,"threads_started":1}
I20260925 14:53:07.610836 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.592502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:52539)
0925 14:53:07.593197 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:07.593239 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:07.594862 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:07.595590 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:07.595599 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:07.596081 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:07.596642 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:07.596668 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.597751 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.597755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:07.599196 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:07.599211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.600083 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.600090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:07.600398 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:07.601223 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:07.601245 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:07.603371 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:07.607217 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:07.607222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:07.607252 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:07.607633 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:07.608030 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:07.608033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:07.608035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:07.608201 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:07.608873 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:07.608879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:07.609153 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 14:53:07.609363 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 14:53:07.609697 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":266,"threads_started":1}
I20260925 14:53:07.612412 26398 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45591
I20260925 14:53:07.612747 26398 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:07.613426 26398 heartbeater.cc:511] Master 127.17.133.254:45591 requested a full tablet report, sending...
I20260925 14:53:07.615329 26209 ts_manager.cc:194] Registered new tserver with Master: d2d554448c3c405c87887c42a251447e (127.17.133.193:37921)
I20260925 14:53:07.616765 26209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:52539
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:53:07.645349 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45591
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:07.954013 26404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:07.954288 26404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:07.954370 26404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:07.961316 26404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:07.961403 26404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:07.961505 26404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:53:07.970935 26404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:07.972209 26404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:07.973433 26404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:07.986093 26410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:07.986093 26409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:07.986476 26412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:07.986971 26404 server_base.cc:1034] running on GCE node
I20260925 14:53:07.987468 26404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:07.988343 26404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:07.989552 26404 hybrid_clock.cc:648] HybridClock initialized: now 1790347987989480 us; error 62 us; skew 500 ppm
I20260925 14:53:07.989859 26404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:07 dist-test-slave-v5vl krb5kdc[26164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347987, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:07.992872 26404 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:53:07.994319 26404 webserver.cc:460] Webserver started at http://127.17.133.194:43807/ using document root <none> and password file <none>
I20260925 14:53:07.994972 26404 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:07.995105 26404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:07.995397 26404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:07.998154 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "3a1b120920a24c66aeaf5eb99f77c6a7"
format_stamp: "Formatted at 2026-09-25 14:53:07 on dist-test-slave-v5vl"
server_key: "7ade98e38cb743e6904dc8340f2e3ec2"
server_key_iv: "1e22917af11739af02c3fcac2e47d12d"
server_key_version: "encryptionkey@0"
I20260925 14:53:07.998878 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "3a1b120920a24c66aeaf5eb99f77c6a7"
format_stamp: "Formatted at 2026-09-25 14:53:07 on dist-test-slave-v5vl"
server_key: "7ade98e38cb743e6904dc8340f2e3ec2"
server_key_iv: "1e22917af11739af02c3fcac2e47d12d"
server_key_version: "encryptionkey@0"
I20260925 14:53:08.003728 26404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 14:53:08.007778 26419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:08.009258 26404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:08.009403 26404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "3a1b120920a24c66aeaf5eb99f77c6a7"
format_stamp: "Formatted at 2026-09-25 14:53:07 on dist-test-slave-v5vl"
server_key: "7ade98e38cb743e6904dc8340f2e3ec2"
server_key_iv: "1e22917af11739af02c3fcac2e47d12d"
server_key_version: "encryptionkey@0"
I20260925 14:53:08.009615 26404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:08.046013 26404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:08.049878 26404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:08.050109 26404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:08.050940 26404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:08.052124 26404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:08.052191 26404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:08.052274 26404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:08.052318 26404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:08.083945 26404 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:35157
I20260925 14:53:08.083992 26531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:35157 every 8 connection(s)
I20260925 14:53:08.085232 26404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:08.093286 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 26404
I20260925 14:53:08.093478 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:08.093905 17943 external_mini_cluster.cc:1428] Setting key 50f4b2c9a69d69ccba67e21e250414e8
Sep 25 14:53:08 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347987, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:53:08.105901 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.087629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:54075 (local address 127.17.133.254:45591)
0925 14:53:08.087789 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:08.087794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:08.088923 (+ 1129us) server_negotiation.cc:407] Connection header received
0925 14:53:08.090683 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:08.090689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:08.090773 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:08.090892 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:08.092452 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.093528 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.095354 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.095813 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.095936 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:08.099295 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:08.099304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:08.099319 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:08.099362 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:08.102875 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.103738 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.103741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.103743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.103851 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.104293 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.104296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.104298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.104551 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:08.104753 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:08.105529 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 14:53:08.105681 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:53:08.107360 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.088051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:54075)
0925 14:53:08.088766 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:08.088804 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:08.090412 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:08.091122 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:08.091130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:08.091610 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:08.092242 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:08.092261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.093673 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.093678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:08.095232 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:08.095244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.096013 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.096019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:08.096263 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:08.097033 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:08.097058 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:08.099093 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:08.103011 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:08.103020 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:08.103039 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:08.103627 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:08.103964 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:08.103971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:08.103978 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:08.104194 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:08.104728 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:08.104739 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:08.105164 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 14:53:08.105461 (+ 297us) client_negotiation.cc:240] Negotiation successful
0925 14:53:08.106000 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":267,"threads_started":1}
I20260925 14:53:08.108757 26532 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45591
I20260925 14:53:08.109055 26532 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:08.109719 26532 heartbeater.cc:511] Master 127.17.133.254:45591 requested a full tablet report, sending...
I20260925 14:53:08.111079 26209 ts_manager.cc:194] Registered new tserver with Master: 3a1b120920a24c66aeaf5eb99f77c6a7 (127.17.133.194:35157)
I20260925 14:53:08.111980 26209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:54075
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:53:08.141607 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45591
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:08.464035 26537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:08.464293 26537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:08.464497 26537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:08.471526 26537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:08.471617 26537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:08.471751 26537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:53:08.481040 26537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:08.482230 26537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:08.483481 26537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:08.495980 26543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:08.496018 26545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:08.496071 26542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:08.496796 26537 server_base.cc:1034] running on GCE node
I20260925 14:53:08.497319 26537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:08.498153 26537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:08.499366 26537 hybrid_clock.cc:648] HybridClock initialized: now 1790347988499334 us; error 45 us; skew 500 ppm
I20260925 14:53:08.499655 26537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:08 dist-test-slave-v5vl krb5kdc[26164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347988, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:08.502626 26537 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:53:08.503945 26537 webserver.cc:460] Webserver started at http://127.17.133.195:45901/ using document root <none> and password file <none>
I20260925 14:53:08.504551 26537 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:08.504717 26537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:08.504962 26537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:08.507647 26537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "73c26d699ffb49109d6105a96c8a94c4"
format_stamp: "Formatted at 2026-09-25 14:53:08 on dist-test-slave-v5vl"
server_key: "50cac8f6cd3465e53143ac553e5a0a21"
server_key_iv: "d51e1d4c9b01766f87dfca4ef56f6bda"
server_key_version: "encryptionkey@0"
I20260925 14:53:08.508363 26537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "73c26d699ffb49109d6105a96c8a94c4"
format_stamp: "Formatted at 2026-09-25 14:53:08 on dist-test-slave-v5vl"
server_key: "50cac8f6cd3465e53143ac553e5a0a21"
server_key_iv: "d51e1d4c9b01766f87dfca4ef56f6bda"
server_key_version: "encryptionkey@0"
I20260925 14:53:08.513268 26537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 14:53:08.516482 26552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:08.517961 26537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:53:08.518155 26537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "73c26d699ffb49109d6105a96c8a94c4"
format_stamp: "Formatted at 2026-09-25 14:53:08 on dist-test-slave-v5vl"
server_key: "50cac8f6cd3465e53143ac553e5a0a21"
server_key_iv: "d51e1d4c9b01766f87dfca4ef56f6bda"
server_key_version: "encryptionkey@0"
I20260925 14:53:08.518366 26537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:08.529845 26537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:08.533260 26537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:08.533524 26537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:08.534405 26537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:08.535524 26537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:08.535591 26537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:08.535737 26537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:08.535781 26537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:08.566897 26537 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:40815
I20260925 14:53:08.566915 26664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:40815 every 8 connection(s)
I20260925 14:53:08.568121 26537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:08.570266 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 26537
I20260925 14:53:08.570451 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:08.570945 17943 external_mini_cluster.cc:1428] Setting key 7ae0e2dce71e4fcf1b69867f1470200b
Sep 25 14:53:08 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347988, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:53:08.587201 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.570486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:34707 (local address 127.17.133.254:45591)
0925 14:53:08.570665 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:08.570671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:08.571965 (+ 1294us) server_negotiation.cc:407] Connection header received
0925 14:53:08.574222 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:08.574227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:08.574338 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:08.574452 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:08.575800 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.576465 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.577941 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.578333 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.578462 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:08.581739 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:08.581744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:08.581747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:08.581777 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:08.584104 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.584969 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.584974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.584976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.585101 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.585578 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.585582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.585585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.585944 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:08.586094 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:08.586819 (+ 725us) server_negotiation.cc:299] Negotiation successful
0925 14:53:08.586968 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:53:08.588143 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.571078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:34707)
0925 14:53:08.571804 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:08.571850 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:08.573983 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:08.574670 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:08.574678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:08.575138 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:08.575644 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:08.575660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.576620 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.576626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:08.577796 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:08.577807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.578539 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.578549 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:08.578787 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:08.579439 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:08.579460 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:08.581571 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:08.584243 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:08.584249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:08.584260 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:08.584829 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:08.585213 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:08.585217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:08.585221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:08.585451 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:08.586092 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:08.586099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:08.586398 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 14:53:08.586624 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 14:53:08.587043 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":278,"threads_started":1}
I20260925 14:53:08.589481 26665 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45591
I20260925 14:53:08.589790 26665 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:08.590423 26665 heartbeater.cc:511] Master 127.17.133.254:45591 requested a full tablet report, sending...
I20260925 14:53:08.591765 26209 ts_manager.cc:194] Registered new tserver with Master: 73c26d699ffb49109d6105a96c8a94c4 (127.17.133.195:40815)
I20260925 14:53:08.592738 26209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:34707
I20260925 14:53:08.597926 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.193@KRBTEST.COM" created.
I20260925 14:53:08.619309 26398 heartbeater.cc:503] Master 127.17.133.254:45591 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab.
Entry for principal oryx/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab.
WARNING: no policy specified for oryx/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab.
Entry for principal oryx/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab.
WARNING: no policy specified for oryx/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab.
Entry for principal oryx/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab.
I20260925 14:53:08.713296 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 26270
Sep 25 14:53:08 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:53:08.729094 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.714395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.17.133.193:37921)
0925 14:53:08.715165 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:08.715172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:08.715197 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:53:08.715255 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:08.715261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:08.715473 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:08.715596 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:08.716900 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.717812 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.718877 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.719439 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.719526 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:08.723445 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:08.723455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:08.723469 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:08.723502 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:08.726386 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.727067 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.727074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.727076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.727188 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.727530 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.727536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.727539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.727915 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:08.728210 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:08.728471 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:53:08.728765 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":455,"threads_started":1}
I20260925 14:53:08.730290 26393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:08.809713 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 26270
I20260925 14:53:08.825632 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 26404
Sep 25 14:53:08 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:53:08.840863 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.826574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.17.133.194:35157)
0925 14:53:08.827061 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:08.827067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:08.827099 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:53:08.827154 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:08.827160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:08.827356 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:08.827475 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:08.828740 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.829682 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.830796 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.831308 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.831398 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:08.835228 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:08.835238 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:08.835250 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:08.835284 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:08.838018 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.838665 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.838672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.838674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.838788 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.839155 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.839162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.839165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.839468 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:08.839751 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:08.840094 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 14:53:08.840337 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
I20260925 14:53:08.842116 26527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:08.923300 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 26404
I20260925 14:53:08.938880 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 26537
Sep 25 14:53:08 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:53:08.954916 26679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.939819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.17.133.195:40815)
0925 14:53:08.940371 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:08.940378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:08.940407 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:53:08.940463 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:08.940470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:08.940753 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:08.940880 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:08.942083 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.943091 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.944332 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:08.944989 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:08.945089 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:08.949249 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:08.949260 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:08.949274 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:08.949309 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:08.952074 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.952832 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.952839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.952841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.952983 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:08.953368 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:08.953374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:08.953378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:08.953710 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:08.954022 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:08.954319 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 14:53:08.954580 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":269,"threads_started":1}
I20260925 14:53:08.956112 26660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:09.043751 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 26537
I20260925 14:53:09.059100 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 26177
I20260925 14:53:09.060181 26239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:09.158646 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 26177
I20260925 14:53:09.181216 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45591
--webserver_interface=127.17.133.254
--webserver_port=33291
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:09.492657 26682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:09.492973 26682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:09.493119 26682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:09.499934 26682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:09.500087 26682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:09.500195 26682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:09.500301 26682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:09.500382 26682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:09.509959 26682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:45591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=33291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:09.511300 26682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:09.512598 26682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:09.524391 26688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:09.524531 26690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:09.524857 26687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:09.525799 26682 server_base.cc:1034] running on GCE node
I20260925 14:53:09.526611 26682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:09.528069 26682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:09.529305 26682 hybrid_clock.cc:648] HybridClock initialized: now 1790347989529242 us; error 72 us; skew 500 ppm
I20260925 14:53:09.529687 26682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:09 dist-test-slave-v5vl krb5kdc[26164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347989, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:09.532699 26682 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:53:09.533989 26682 webserver.cc:460] Webserver started at http://127.17.133.254:33291/ using document root <none> and password file <none>
I20260925 14:53:09.534608 26682 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:09.534708 26682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:09.539706 26682 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 14:53:09.542248 26697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:09.543399 26682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:09.543581 26682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "a4483e17140f4b719b0159f7484875b4"
format_stamp: "Formatted at 2026-09-25 14:53:06 on dist-test-slave-v5vl"
server_key: "49b97de04525b0f89365a67fc9ddaec5"
server_key_iv: "6a7f706c41d5203357ddeed001a46d43"
server_key_version: "encryptionkey@0"
I20260925 14:53:09.544472 26682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:09.582814 26682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:09.586979 26682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:09.587242 26682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:09.605360 26682 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:45591
I20260925 14:53:09.605389 26748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:45591 every 8 connection(s)
I20260925 14:53:09.606822 26682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:09.611521 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 26682
I20260925 14:53:09.612243 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:37921
--local_ip_for_outbound_sockets=127.17.133.193
--tserver_master_addrs=127.17.133.254:45591
--webserver_port=35585
--webserver_interface=127.17.133.193
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.133.193
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:09.613643 26749 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:53:09.616771 26749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: Bootstrap starting.
I20260925 14:53:09.642614 26749 log.cc:824] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:09.653324 26749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:53:09.653924 26749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: Bootstrap complete.
I20260925 14:53:09.666947 26749 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } }
I20260925 14:53:09.668095 26749 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4483e17140f4b719b0159f7484875b4, State: Initialized, Role: FOLLOWER
I20260925 14:53:09.668852 26749 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } }
I20260925 14:53:09.669167 26749 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:09.669307 26749 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:09.669487 26749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:53:09.674993 26749 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } }
I20260925 14:53:09.675638 26749 leader_election.cc:302] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4483e17140f4b719b0159f7484875b4; no voters:
I20260925 14:53:09.676339 26749 leader_election.cc:288] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:53:09.676461 26753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:53:09.678006 26753 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [term 2 LEADER]: Becoming Leader. State: Replica: a4483e17140f4b719b0159f7484875b4, State: Running, Role: LEADER
I20260925 14:53:09.678509 26753 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } }
I20260925 14:53:09.678768 26749 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:09.681037 26755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4483e17140f4b719b0159f7484875b4. Latest consensus state: current_term: 2 leader_uuid: "a4483e17140f4b719b0159f7484875b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } } }
I20260925 14:53:09.681074 26754 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a4483e17140f4b719b0159f7484875b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4483e17140f4b719b0159f7484875b4" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45591 } } }
I20260925 14:53:09.681406 26755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:09.681445 26754 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:09.694051 26758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:09.699496 26758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:09.700836 26758 catalog_manager.cc:1259] Loaded cluster ID: f286c41257eb467faf56c38f476e24d7
I20260925 14:53:09.700946 26758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:09.705366 26758 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:09.706707 26758 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a4483e17140f4b719b0159f7484875b4: Loaded TSK: 0
I20260925 14:53:09.708114 26758 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:53:09.968477 26751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:09.968838 26751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:09.969017 26751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:09.976182 26751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:09.976275 26751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:09.976436 26751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:53:09.986162 26751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.193.keytab
--principal=oryx/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:37921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=35585
--tserver_master_addrs=127.17.133.254:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:09.987435 26751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:09.988773 26751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:10.002936 26776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:10.002965 26775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:10.003588 26778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:10.004714 26751 server_base.cc:1034] running on GCE node
I20260925 14:53:10.005375 26751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:10.006389 26751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:10.007611 26751 hybrid_clock.cc:648] HybridClock initialized: now 1790347990007550 us; error 55 us; skew 500 ppm
I20260925 14:53:10.007979 26751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347990, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:10.011179 26751 init.cc:377] Logged in from keytab as oryx/127.17.133.193@KRBTEST.COM (short username oryx)
I20260925 14:53:10.012787 26751 webserver.cc:460] Webserver started at http://127.17.133.193:35585/ using document root <none> and password file <none>
I20260925 14:53:10.013477 26751 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:10.013653 26751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:10.019182 26751 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 14:53:10.022004 26785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.023468 26751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:10.023648 26751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "d2d554448c3c405c87887c42a251447e"
format_stamp: "Formatted at 2026-09-25 14:53:07 on dist-test-slave-v5vl"
server_key: "12476bcbce1c4b87b6e81c2cc874e58d"
server_key_iv: "a841e2d6a49760fe0ce500cee4bf9c80"
server_key_version: "encryptionkey@0"
I20260925 14:53:10.024572 26751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:10.043527 26751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:10.047713 26751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:10.047971 26751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:10.049008 26751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:10.050295 26751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:10.050419 26751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.050544 26751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:10.050650 26751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.083078 26751 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:37921
I20260925 14:53:10.083173 26897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:37921 every 8 connection(s)
I20260925 14:53:10.084477 26751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:10.086689 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 26751
I20260925 14:53:10.087286 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:35157
--local_ip_for_outbound_sockets=127.17.133.194
--tserver_master_addrs=127.17.133.254:45591
--webserver_port=43807
--webserver_interface=127.17.133.194
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.133.194
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:53:10.104954 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:10.087386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:49053 (local address 127.17.133.254:45591)
0925 14:53:10.088184 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:10.088202 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:10.089034 (+ 832us) server_negotiation.cc:407] Connection header received
0925 14:53:10.092168 (+ 3134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:10.092186 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:10.092677 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:10.092978 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:10.094911 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.096083 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.097642 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.098264 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.098458 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.103910 (+ 5452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:49053: BlockingRecv error: recv got EOF from 127.17.133.193:49053 (error 108)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":282,"threads_started":1}
I20260925 14:53:10.105029 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.087473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:49053)
0925 14:53:10.088835 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.088889 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:10.091535 (+ 2646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:10.093172 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:10.093185 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:10.093943 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:10.094703 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.094723 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.096241 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.096246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.097493 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.097504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.098492 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.098499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.098752 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.099523 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:10.099544 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:10.103669 (+ 4125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":844,"threads_started":1}
W20260925 14:53:10.105713 26898 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:45591 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.133.254:45591: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:10.113464 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.106399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:60633)
0925 14:53:10.106576 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.106594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:10.106694 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:10.106992 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:10.106997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:10.107298 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:10.107422 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.107429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.108354 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.108359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.109273 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.109283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.109939 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.109945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.110104 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.110645 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:10.110660 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:10.113227 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 14:53:10.113617 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:10.106458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:60633 (local address 127.17.133.254:45591)
0925 14:53:10.106607 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:10.106614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:10.106634 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:53:10.106803 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:10.106808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:10.106876 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:10.106972 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:10.107547 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.108212 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.109403 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.109807 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.109928 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.113425 (+ 3497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:60633: BlockingRecv error: recv got EOF from 127.17.133.193:60633 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:10.121465 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.114054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:57321)
0925 14:53:10.114201 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.114220 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:10.114312 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:10.114702 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:10.114705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:10.114959 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:10.115079 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.115086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.116373 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.116376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.117321 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.117331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.118170 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.118185 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.118304 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.118853 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:10.118867 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:10.121276 (+ 2409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:53:10.121729 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:10.114275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:57321 (local address 127.17.133.254:45591)
0925 14:53:10.114436 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:10.114442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:10.114462 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:53:10.114508 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:10.114513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:10.114584 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:10.114678 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:10.115192 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.116224 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.117450 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.118048 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.118160 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.121503 (+ 3343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:57321: BlockingRecv error: recv got EOF from 127.17.133.193:57321 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 14:53:10.121733 26898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:53:10.422219 26902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:10.422705 26902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:10.422894 26902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:10.437467 26902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:10.437614 26902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:10.437850 26902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:53:10.449179 26902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.194.keytab
--principal=oryx/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:35157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=43807
--tserver_master_addrs=127.17.133.254:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:10.450405 26902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:10.451632 26902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:10.464668 26907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:10.464564 26908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:10.464614 26910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:10.465544 26902 server_base.cc:1034] running on GCE node
I20260925 14:53:10.466110 26902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:10.467007 26902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:10.468194 26902 hybrid_clock.cc:648] HybridClock initialized: now 1790347990468156 us; error 32 us; skew 500 ppm
I20260925 14:53:10.468497 26902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347990, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:10.471647 26902 init.cc:377] Logged in from keytab as oryx/127.17.133.194@KRBTEST.COM (short username oryx)
I20260925 14:53:10.473083 26902 webserver.cc:460] Webserver started at http://127.17.133.194:43807/ using document root <none> and password file <none>
I20260925 14:53:10.473747 26902 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:10.473896 26902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:10.478906 26902 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 14:53:10.481555 26917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.482784 26902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:53:10.482976 26902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "3a1b120920a24c66aeaf5eb99f77c6a7"
format_stamp: "Formatted at 2026-09-25 14:53:07 on dist-test-slave-v5vl"
server_key: "7ade98e38cb743e6904dc8340f2e3ec2"
server_key_iv: "1e22917af11739af02c3fcac2e47d12d"
server_key_version: "encryptionkey@0"
I20260925 14:53:10.483872 26902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:10.503352 26902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:10.508242 26902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:10.508451 26902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:10.509357 26902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:10.510571 26902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:10.510668 26902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.510744 26902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:10.510798 26902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.543143 26902 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:35157
I20260925 14:53:10.543196 27029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:35157 every 8 connection(s)
I20260925 14:53:10.544574 26902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:10.550551 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 26902
I20260925 14:53:10.551126 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:40815
--local_ip_for_outbound_sockets=127.17.133.195
--tserver_master_addrs=127.17.133.254:45591
--webserver_port=45901
--webserver_interface=127.17.133.195
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.133.195
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:53:10.559801 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:10.547053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:48711 (local address 127.17.133.254:45591)
0925 14:53:10.547224 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:10.547228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:10.548442 (+ 1214us) server_negotiation.cc:407] Connection header received
0925 14:53:10.550346 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:10.550351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:10.550439 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:10.550550 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:10.552083 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.552853 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.554357 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.554763 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.554893 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.559570 (+ 4677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:48711: BlockingRecv error: recv got EOF from 127.17.133.194:48711 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:53:10.560501 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.547513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:48711)
0925 14:53:10.548277 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.548336 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:10.550087 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:10.550788 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:10.550797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:10.551347 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:10.551918 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.551935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.553002 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.553009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.554213 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.554224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.554987 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.554994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.555219 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.555861 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:10.555886 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:10.559399 (+ 3513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":248,"threads_started":1}
W20260925 14:53:10.561113 27030 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:45591 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.133.254:45591: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:10.569630 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.561693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:55065)
0925 14:53:10.561868 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.561882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:10.561952 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:10.562271 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:10.562274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:10.562535 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:10.562648 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.562655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.564017 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.564020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.564955 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.564974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.565980 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.565984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.566102 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.566676 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:10.566692 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:10.569419 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 14:53:10.570183 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:10.561786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:55065 (local address 127.17.133.254:45591)
0925 14:53:10.561988 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:10.561996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:10.562016 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:53:10.562065 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:10.562072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:10.562152 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:10.562249 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:10.562778 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.563897 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.565090 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.565830 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.565994 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.569723 (+ 3729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:55065: BlockingRecv error: recv got EOF from 127.17.133.194:55065 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:10.577553 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.570293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:47837)
0925 14:53:10.570627 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.570649 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:10.570762 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:10.571062 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:10.571066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:10.571384 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:10.571512 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.571519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.572402 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.572407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.573325 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.573335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.574001 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.574007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.574195 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.574734 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:10.574749 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:10.577285 (+ 2536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197}
W20260925 14:53:10.577787 27030 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:53:10.577936 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:10.570373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:47837 (local address 127.17.133.254:45591)
0925 14:53:10.570578 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:10.570585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:10.570817 (+ 232us) server_negotiation.cc:407] Connection header received
0925 14:53:10.570881 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:10.570884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:10.570935 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:10.571033 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:10.571637 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.572275 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.573454 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.573855 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.573974 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.577447 (+ 3473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:47837: BlockingRecv error: recv got EOF from 127.17.133.194:47837 (error 108)
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":336,"threads_started":1}
W20260925 14:53:10.866499 27033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:10.866787 27033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:10.866961 27033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:10.873809 27033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:10.873958 27033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:10.874157 27033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:53:10.883665 27033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/krb5kdc/oryx_127.17.133.195.keytab
--principal=oryx/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:40815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=45901
--tserver_master_addrs=127.17.133.254:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:10.885057 27033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:10.886308 27033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:10.899093 27039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:10.899109 27042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:10.899048 27040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:10.899760 27033 server_base.cc:1034] running on GCE node
I20260925 14:53:10.900280 27033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:10.901275 27033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:10.902490 27033 hybrid_clock.cc:648] HybridClock initialized: now 1790347990902456 us; error 42 us; skew 500 ppm
I20260925 14:53:10.902801 27033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347990, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:10.905942 27033 init.cc:377] Logged in from keytab as oryx/127.17.133.195@KRBTEST.COM (short username oryx)
I20260925 14:53:10.907310 27033 webserver.cc:460] Webserver started at http://127.17.133.195:45901/ using document root <none> and password file <none>
I20260925 14:53:10.907949 27033 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:10.908128 27033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:10.913347 27033 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:53:10.915831 27049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.917106 27033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:10.917275 27033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "73c26d699ffb49109d6105a96c8a94c4"
format_stamp: "Formatted at 2026-09-25 14:53:08 on dist-test-slave-v5vl"
server_key: "50cac8f6cd3465e53143ac553e5a0a21"
server_key_iv: "d51e1d4c9b01766f87dfca4ef56f6bda"
server_key_version: "encryptionkey@0"
I20260925 14:53:10.918195 27033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:10.946486 27033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:10.950455 27033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:10.950706 27033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:10.951591 27033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:10.952809 27033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:10.952926 27033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.953047 27033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:10.953123 27033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.984076 27033 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:40815
I20260925 14:53:10.984157 27161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:40815 every 8 connection(s)
I20260925 14:53:10.985570 27033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:10.992547 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 27033
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:10 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:53:11.001677 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:10.988107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:49443 (local address 127.17.133.254:45591)
0925 14:53:10.988304 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:10.988309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:10.989534 (+ 1225us) server_negotiation.cc:407] Connection header received
0925 14:53:10.991240 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:10.991244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:10.991309 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:10.991393 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:10.992879 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.993836 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.995856 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.996423 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.996560 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.001376 (+ 4816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:49443: BlockingRecv error: recv got EOF from 127.17.133.195:49443 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 14:53:11.002262 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.988728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:49443)
0925 14:53:10.989372 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.989415 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:10.991003 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:10.991600 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:10.991607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:10.992084 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:10.992701 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.992725 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.994052 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.994059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.995712 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:10.995723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.996677 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.996684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:10.996963 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:10.997635 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:10.997657 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:11.001156 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":248,"threads_started":1}
W20260925 14:53:11.002889 27162 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:45591 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.133.254:45591: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
I20260925 14:53:11.008334 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.993517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.17.133.254:45591)
0925 14:53:10.993715 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:10.993720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:10.994066 (+ 346us) server_negotiation.cc:407] Connection header received
0925 14:53:10.994288 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:10.994295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:10.994378 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:10.994501 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:10.996152 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.997148 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.998533 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:10.998961 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:10.999110 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.001821 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:11.001833 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:11.001852 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:11.001899 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:11.005123 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:11.005862 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:11.005868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:11.005874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:11.006054 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:11.006442 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:11.006447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:11.006451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:11.006876 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:11.007214 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:11.007585 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 14:53:11.007924 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 14:53:11 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:11 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:11.011526 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:11.003386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:44819)
0925 14:53:11.003821 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:11.003836 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:11.003951 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:11.004288 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:11.004293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:11.004700 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:11.004826 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:11.004833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.005943 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.005949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:11.006903 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:11.006913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.007607 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.007614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:11.007788 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.008334 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:11.008350 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:11.011277 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 14:53:11.011740 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:11.003536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:44819 (local address 127.17.133.254:45591)
0925 14:53:11.003691 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:11.003697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:11.003901 (+ 204us) server_negotiation.cc:407] Connection header received
0925 14:53:11.004093 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:11.004097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:11.004146 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:11.004255 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:11.004951 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.005817 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.007041 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.007457 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.007583 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.011478 (+ 3895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:44819: BlockingRecv error: recv got EOF from 127.17.133.195:44819 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 14:53:11 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:11 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:11.018946 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:11.012477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:33889)
0925 14:53:11.012678 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:11.012699 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:11.012777 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:11.013048 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:11.013051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:11.013282 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:11.013408 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:11.013415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.014294 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.014297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:11.015177 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:11.015187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.015785 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.015789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:11.015898 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.016346 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:11.016361 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:11.018750 (+ 2389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 14:53:11.019127 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:11.012559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:33889 (local address 127.17.133.254:45591)
0925 14:53:11.012802 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:11.012805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:11.012834 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:53:11.012869 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:11.012872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:11.012936 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:11.013002 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:11.013514 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.014186 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.015285 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.015677 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.015760 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.018916 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:33889: BlockingRecv error: recv got EOF from 127.17.133.195:33889 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
W20260925 14:53:11.019245 27162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 14:53:11 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:11 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:11.130155 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:11.122225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:59257)
0925 14:53:11.122703 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:11.122722 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:11.122867 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:11.123265 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:11.123269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:11.123819 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:11.124009 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:11.124018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.124984 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.124987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:11.125901 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:11.125911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.126525 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.126529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:11.126638 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.127223 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:11.127251 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:11.129940 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":218,"threads_started":1}
W20260925 14:53:11.130313 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:11.122277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:59257 (local address 127.17.133.254:45591)
0925 14:53:11.122467 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:11.122475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:11.122820 (+ 345us) server_negotiation.cc:407] Connection header received
0925 14:53:11.123020 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:11.123028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:11.123114 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:11.123219 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:11.124130 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.124874 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.126007 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.126395 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.126477 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.130110 (+ 3633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:59257: BlockingRecv error: recv got EOF from 127.17.133.193:59257 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 14:53:11 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:11 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:11.586485 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:11.578290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:53523)
0925 14:53:11.578791 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:11.578806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:11.578913 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:11.579182 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:11.579185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:11.579677 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:11.579871 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:11.579880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.580864 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.580867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:11.581770 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:11.581782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.582486 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.582490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:11.582601 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.583205 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:11.583222 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:11.586249 (+ 3027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":295,"threads_started":1}
W20260925 14:53:11.586655 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:11.578312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:53523 (local address 127.17.133.254:45591)
0925 14:53:11.578496 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:11.578518 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:11.578886 (+ 368us) server_negotiation.cc:407] Connection header received
0925 14:53:11.579017 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:11.579020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:11.579074 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:11.579146 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:11.579987 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.580740 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.581916 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.582334 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.582433 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.586445 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:53523: BlockingRecv error: recv got EOF from 127.17.133.194:53523 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 14:53:12 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:12 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:12.028795 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:12.020056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:52507)
0925 14:53:12.020602 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:12.020621 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:12.020756 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:12.021129 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:12.021134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:12.021688 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:12.021983 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:12.021995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.023120 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.023127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:12.024079 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:12.024090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.024846 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.024852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:12.025026 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:12.025636 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:12.025653 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:12.028473 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":302,"threads_started":1}
W20260925 14:53:12.028923 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:12.020129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:52507 (local address 127.17.133.254:45591)
0925 14:53:12.020299 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:12.020303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:12.020702 (+ 399us) server_negotiation.cc:407] Connection header received
0925 14:53:12.020900 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:12.020905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:12.020975 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:12.021083 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:12.022112 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.022993 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.024215 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.024679 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.024819 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:12.028695 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:52507: BlockingRecv error: recv got EOF from 127.17.133.195:52507 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 14:53:12 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:12 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:12.138779 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:12.130980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:34617)
0925 14:53:12.131535 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:12.131552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:12.131661 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:12.131946 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:12.131949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:12.132311 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:12.132473 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:12.132481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.133525 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.133529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:12.134419 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:12.134430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.135073 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.135077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:12.135184 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:12.135719 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:12.135735 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:12.138413 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":319,"threads_started":1}
W20260925 14:53:12.138836 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:12.131089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:34617 (local address 127.17.133.254:45591)
0925 14:53:12.131252 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:12.131286 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:12.131624 (+ 338us) server_negotiation.cc:407] Connection header received
0925 14:53:12.131759 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:12.131763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:12.131822 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:12.131905 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:12.132598 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.133403 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.134531 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.134943 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.135038 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:12.138608 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:34617: BlockingRecv error: recv got EOF from 127.17.133.193:34617 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 14:53:12 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:12 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:12.595989 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:12.587640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:33387)
0925 14:53:12.588274 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:12.588290 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:12.588450 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:12.588917 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:12.588922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:12.589402 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:12.589566 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:12.589575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.590548 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.590555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:12.591432 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:12.591442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.592132 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.592137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:12.592358 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:12.592989 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:12.593007 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:12.595649 (+ 2642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":411,"threads_started":1}
W20260925 14:53:12.596063 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:12.587694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:33387 (local address 127.17.133.254:45591)
0925 14:53:12.587835 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:12.587840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:12.588377 (+ 537us) server_negotiation.cc:407] Connection header received
0925 14:53:12.588716 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:12.588719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:12.588774 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:12.588878 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:12.589706 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.590416 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.591574 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:12.591983 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:12.592105 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:12.595835 (+ 3730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:33387: BlockingRecv error: recv got EOF from 127.17.133.194:33387 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 14:53:13 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:13 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:13.038156 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:13.029694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:45243)
0925 14:53:13.030213 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:13.030233 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:13.030355 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:13.030668 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:13.030672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:13.031046 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:13.031211 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:13.031219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.032134 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.032138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:13.033256 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:13.033272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.034190 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.034196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:13.034358 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:13.035068 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:13.035086 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:13.037810 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
W20260925 14:53:13.038273 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:13.029909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:45243 (local address 127.17.133.254:45591)
0925 14:53:13.030084 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:13.030088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:13.030315 (+ 227us) server_negotiation.cc:407] Connection header received
0925 14:53:13.030467 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:13.030470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:13.030541 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:13.030626 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:13.031324 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.032017 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.033424 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.034012 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.034147 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:13.038014 (+ 3867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:45243: BlockingRecv error: recv got EOF from 127.17.133.195:45243 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 14:53:13 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:13 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:13.147394 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:13.139621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:50693)
0925 14:53:13.140082 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:13.140102 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:13.140247 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:13.140663 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:13.140670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:13.141001 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:13.141233 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:13.141243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.142220 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.142224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:13.143114 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:13.143124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.143699 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.143703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:13.143817 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:13.144434 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:13.144451 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:13.147089 (+ 2638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":249,"threads_started":1}
W20260925 14:53:13.147634 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:13.139686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:50693 (local address 127.17.133.254:45591)
0925 14:53:13.139853 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:13.139860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:13.140187 (+ 327us) server_negotiation.cc:407] Connection header received
0925 14:53:13.140412 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:13.140416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:13.140500 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:13.140627 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:13.141359 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.142096 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.143228 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.143588 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.143673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:13.147308 (+ 3635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:50693: BlockingRecv error: recv got EOF from 127.17.133.193:50693 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 14:53:13 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:13 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:13.604825 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:13.597038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:60061)
0925 14:53:13.597583 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:13.597598 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:13.597709 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:13.598043 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:13.598047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:13.598454 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:13.598668 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:13.598680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.599704 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.599708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:13.600612 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:13.600623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.601201 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.601205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:13.601317 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:13.601878 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:13.601893 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:13.604430 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":327,"threads_started":1}
W20260925 14:53:13.604907 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:13.597122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:60061 (local address 127.17.133.254:45591)
0925 14:53:13.597349 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:13.597355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:13.597677 (+ 322us) server_negotiation.cc:407] Connection header received
0925 14:53:13.597797 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:13.597802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:13.597873 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:13.597985 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:13.598815 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.599584 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.600716 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.601090 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.601176 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:13.604662 (+ 3486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:60061: BlockingRecv error: recv got EOF from 127.17.133.194:60061 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 14:53:14 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:14 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:14.047662 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:14.039177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:52065)
0925 14:53:14.039612 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:14.039629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:14.039778 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:14.040154 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:14.040159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:14.040479 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:14.040661 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:14.040669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.041958 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.041967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:14.043109 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:14.043120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.043813 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.043818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:14.043990 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:14.044616 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:14.044633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:14.047295 (+ 2662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
W20260925 14:53:14.047731 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:14.039259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:52065 (local address 127.17.133.254:45591)
0925 14:53:14.039462 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:14.039469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:14.039710 (+ 241us) server_negotiation.cc:407] Connection header received
0925 14:53:14.039908 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:14.039912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:14.040020 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:14.040114 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:14.040811 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.041813 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.043269 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.043673 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.043791 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:14.047479 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:52065: BlockingRecv error: recv got EOF from 127.17.133.195:52065 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 14:53:14 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:14 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:14.156200 27175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:14.148243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:50863)
0925 14:53:14.148790 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:14.148807 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:14.148969 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:14.149293 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:14.149297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:14.149653 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:14.149797 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:14.149805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.150782 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.150789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:14.151738 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:14.151749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.152426 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.152432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:14.152658 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:14.153260 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:14.153276 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:14.155873 (+ 2597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":298,"threads_started":1}
W20260925 14:53:14.156317 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:14.148353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:50863 (local address 127.17.133.254:45591)
0925 14:53:14.148566 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:14.148574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:14.148901 (+ 327us) server_negotiation.cc:407] Connection header received
0925 14:53:14.149096 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:14.149099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:14.149155 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:14.149261 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:14.149951 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.150658 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.151879 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.152286 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.152416 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:14.156073 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:50863: BlockingRecv error: recv got EOF from 127.17.133.193:50863 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 14:53:14 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:14 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:14.614689 27176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:14.605830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:47355)
0925 14:53:14.606291 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:14.606307 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:14.606454 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:14.606793 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:14.606798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:14.607393 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:14.607592 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:14.607606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.608937 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.608944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:14.609897 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:14.609908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.610702 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.610707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:14.610892 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:14.611571 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:14.611588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:14.614372 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
W20260925 14:53:14.614769 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:14.605867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:47355 (local address 127.17.133.254:45591)
0925 14:53:14.606061 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:14.606070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:14.606390 (+ 320us) server_negotiation.cc:407] Connection header received
0925 14:53:14.606594 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:14.606598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:14.606658 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:14.606761 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:14.607720 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.608816 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.610045 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:14.610540 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:14.610704 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:14.614544 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:47355: BlockingRecv error: recv got EOF from 127.17.133.194:47355 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 14:53:15 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:15 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:15.057156 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:15.048539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:45193)
0925 14:53:15.049133 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:15.049152 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:15.049299 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:15.049640 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:15.049645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:15.050045 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:15.050183 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:15.050190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.051239 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.051247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:15.052205 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:15.052216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.053015 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.053021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:15.053193 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:15.053848 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:15.053865 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:15.056931 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":293,"threads_started":1}
W20260925 14:53:15.057320 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:15.048632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:45193 (local address 127.17.133.254:45591)
0925 14:53:15.048841 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:15.048849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:15.049234 (+ 385us) server_negotiation.cc:407] Connection header received
0925 14:53:15.049433 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:15.049438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:15.049510 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:15.049611 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:15.050322 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.051082 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.052346 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.052864 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.053005 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:15.057107 (+ 4102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:45193: BlockingRecv error: recv got EOF from 127.17.133.195:45193 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 14:53:15 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:15 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:15.165241 27178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:15.157162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:46793)
0925 14:53:15.157676 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:15.157708 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:15.157815 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:15.158108 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:15.158112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:15.158574 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:15.158733 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:15.158743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.159676 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.159680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:15.160569 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:15.160580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.161253 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.161257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:15.161374 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:15.162065 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:15.162103 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:15.164991 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":278,"threads_started":1}
W20260925 14:53:15.165364 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:15.157256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:46793 (local address 127.17.133.254:45591)
0925 14:53:15.157437 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:15.157441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:15.157783 (+ 342us) server_negotiation.cc:407] Connection header received
0925 14:53:15.157893 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:15.157896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:15.157950 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:15.158027 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:15.158900 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.159563 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.160734 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.161119 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.161249 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:15.165185 (+ 3936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:46793: BlockingRecv error: recv got EOF from 127.17.133.193:46793 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 14:53:15 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:15 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:15.627208 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:15.615618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:50903)
0925 14:53:15.616129 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:15.616145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:15.616251 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:15.616649 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:15.616654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:15.617050 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:15.617205 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:15.617213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.618193 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.618196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:15.619090 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:15.619100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.619745 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.619749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:15.619859 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:15.624223 (+ 4364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:15.624241 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:15.626981 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":258,"threads_started":1}
W20260925 14:53:15.627377 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:15.615769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:50903 (local address 127.17.133.254:45591)
0925 14:53:15.615946 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:15.615950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:15.616217 (+ 267us) server_negotiation.cc:407] Connection header received
0925 14:53:15.616396 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:15.616400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:15.616462 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:15.616548 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:15.617329 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.618079 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.619218 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.619627 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.619718 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:15.627153 (+ 7435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:50903: BlockingRecv error: recv got EOF from 127.17.133.194:50903 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 14:53:16 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:16 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:16.066380 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:16.058399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:33433)
0925 14:53:16.058920 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:16.058939 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:16.059131 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:16.059519 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:16.059523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:16.060002 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:16.060252 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:16.060278 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.061297 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.061300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:16.062168 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:16.062177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.062779 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.062782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:16.062886 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:16.063496 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:16.063512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:16.066149 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":219,"threads_started":1}
W20260925 14:53:16.066526 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:16.058614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:33433 (local address 127.17.133.254:45591)
0925 14:53:16.058824 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:16.058831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:16.059078 (+ 247us) server_negotiation.cc:407] Connection header received
0925 14:53:16.059282 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:16.059287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:16.059365 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:16.059458 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:16.060420 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.061162 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.062273 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.062672 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.062755 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:16.066308 (+ 3553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:33433: BlockingRecv error: recv got EOF from 127.17.133.195:33433 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 14:53:16 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:16 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:16.176391 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:16.166222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:42325)
0925 14:53:16.166733 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:16.166749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:16.166897 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:16.167238 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:16.167243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:16.167651 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:16.167797 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:16.167805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.168850 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.168857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:16.170583 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:16.170601 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.171670 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.171677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:16.171890 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:16.172676 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:16.172703 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:16.176094 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
W20260925 14:53:16.176788 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:16.166293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:42325 (local address 127.17.133.254:45591)
0925 14:53:16.166534 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:16.166542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:16.166833 (+ 291us) server_negotiation.cc:407] Connection header received
0925 14:53:16.167031 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:16.167036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:16.167107 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:16.167208 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:16.167937 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.168726 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.170745 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.171491 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.171638 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:16.176529 (+ 4891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:42325: BlockingRecv error: recv got EOF from 127.17.133.193:42325 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 14:53:16 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:16 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:16.636013 27182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:16.628038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:54507)
0925 14:53:16.628607 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:16.628624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:16.628785 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:16.629106 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:16.629110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:16.629524 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:16.629699 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:16.629708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.630700 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.630706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:16.631647 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:16.631658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.632345 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.632351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:16.632518 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:16.633124 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:16.633141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:16.635699 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":288,"threads_started":1}
W20260925 14:53:16.636276 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:16.628154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:54507 (local address 127.17.133.254:45591)
0925 14:53:16.628384 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:16.628392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:16.628714 (+ 322us) server_negotiation.cc:407] Connection header received
0925 14:53:16.628904 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:16.628908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:16.628977 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:16.629080 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:16.629840 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.630555 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.631790 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:16.632194 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:16.632325 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:16.635991 (+ 3666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:54507: BlockingRecv error: recv got EOF from 127.17.133.194:54507 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 14:53:17 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:17 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:17.075362 27183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:17.067344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:52461)
0925 14:53:17.067798 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:17.067818 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:17.067928 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:17.068213 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:17.068216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:17.068575 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:17.068743 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:17.068751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.069672 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.069675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:17.070576 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:17.070587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.071257 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.071261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:17.071382 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:17.071927 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:17.071943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:17.075056 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":259,"threads_started":1}
W20260925 14:53:17.075481 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:17.067473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:52461 (local address 127.17.133.254:45591)
0925 14:53:17.067693 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:17.067698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:17.067894 (+ 196us) server_negotiation.cc:407] Connection header received
0925 14:53:17.068022 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:17.068026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:17.068084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:17.068160 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:17.068848 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.069566 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.070696 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.071118 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.071201 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:17.075236 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:52461: BlockingRecv error: recv got EOF from 127.17.133.195:52461 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 14:53:17 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:17 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:17.185204 27184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:17.177505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:35995)
0925 14:53:17.177967 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:17.177981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:17.178127 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:17.178432 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:17.178437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:17.178973 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:17.179122 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:17.179130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.180160 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.180165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:17.181004 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:17.181013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.181630 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.181634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:17.181786 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:17.182260 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:17.182275 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:17.184856 (+ 2581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":230,"threads_started":1}
W20260925 14:53:17.185271 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:17.177606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:35995 (local address 127.17.133.254:45591)
0925 14:53:17.177824 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:17.177830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:17.178065 (+ 235us) server_negotiation.cc:407] Connection header received
0925 14:53:17.178252 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:17.178257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:17.178318 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:17.178407 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:17.179253 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.180034 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.181123 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.181504 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.181617 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:17.185042 (+ 3425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:35995: BlockingRecv error: recv got EOF from 127.17.133.193:35995 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 14:53:17 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:17 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:17.645253 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:17.637008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:49813)
0925 14:53:17.637578 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:17.637594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:17.637754 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:17.638112 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:17.638117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:17.638544 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:17.638703 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:17.638712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.639718 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.639725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:17.640693 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:17.640704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.641450 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.641456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:17.641623 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:17.642188 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:17.642205 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:17.644900 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":267,"threads_started":1}
W20260925 14:53:17.645306 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:17.637103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:49813 (local address 127.17.133.254:45591)
0925 14:53:17.637342 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:17.637351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:17.637687 (+ 336us) server_negotiation.cc:407] Connection header received
0925 14:53:17.637899 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:17.637903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:17.637967 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:17.638081 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:17.638855 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.639585 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.640855 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.641304 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.641439 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:17.645068 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:49813: BlockingRecv error: recv got EOF from 127.17.133.194:49813 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 14:53:18 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:18 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:53:18.084273 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:18.076351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:34401 (local address 127.17.133.254:45591)
0925 14:53:18.076558 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:18.076564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:18.076933 (+ 369us) server_negotiation.cc:407] Connection header received
0925 14:53:18.077076 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:18.077079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:18.077135 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:18.077250 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:18.077881 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.078611 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.079753 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.080192 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.080281 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:18.084119 (+ 3838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:34401: BlockingRecv error: recv got EOF from 127.17.133.195:34401 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:53:18.084273 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:18.076262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:34401)
0925 14:53:18.076845 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:18.076865 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:18.076972 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:18.077287 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:18.077293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:18.077635 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:18.077776 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:18.077784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.078773 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.078776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:18.079640 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:18.079649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.080312 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.080316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:18.080431 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:18.080949 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:18.080964 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:18.083955 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":311,"threads_started":1}
Sep 25 14:53:18 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:18 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:18.194360 27187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:18.185995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:33219)
0925 14:53:18.186542 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:18.186558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:18.186706 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:18.187029 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:18.187034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:18.187467 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:18.187627 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:18.187635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.188762 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.188768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:18.190071 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:18.190083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.190775 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.190779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:18.190892 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:18.191450 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:18.191465 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:18.194040 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
W20260925 14:53:18.194461 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:18.186162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:33219 (local address 127.17.133.254:45591)
0925 14:53:18.186396 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:18.186402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:18.186638 (+ 236us) server_negotiation.cc:407] Connection header received
0925 14:53:18.186831 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:18.186836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:18.186905 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:18.187004 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:18.187756 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.188573 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.190183 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.190629 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.190720 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:18.194220 (+ 3500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:33219: BlockingRecv error: recv got EOF from 127.17.133.193:33219 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 14:53:18 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:18 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:18.654451 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:18.646335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:59721)
0925 14:53:18.646922 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:18.646938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:18.647083 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:18.647417 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:18.647422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:18.647927 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:18.648132 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:18.648146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.649455 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.649459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:18.650344 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:18.650353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.651016 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.651020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:18.651126 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:18.651681 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:18.651697 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:18.654239 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":327,"threads_started":1}
W20260925 14:53:18.654551 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:18.646400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:59721 (local address 127.17.133.254:45591)
0925 14:53:18.646610 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:18.646619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:18.647019 (+ 400us) server_negotiation.cc:407] Connection header received
0925 14:53:18.647212 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:18.647218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:18.647289 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:18.647390 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:18.648276 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.649329 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.650449 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:18.650870 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:18.650967 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:18.654393 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:59721: BlockingRecv error: recv got EOF from 127.17.133.194:59721 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 14:53:19 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:19 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:19.093530 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:19.085415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:51461)
0925 14:53:19.085921 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:19.085939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:19.086043 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:19.086391 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:19.086394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:19.086784 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:19.086923 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:19.086930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.088013 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.088020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:19.088990 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:19.089001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.089780 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.089786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:19.089943 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:19.090451 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:19.090467 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:19.093263 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":272,"threads_started":1}
W20260925 14:53:19.093906 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:19.085483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:51461 (local address 127.17.133.254:45591)
0925 14:53:19.085675 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:19.085681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:19.086021 (+ 340us) server_negotiation.cc:407] Connection header received
0925 14:53:19.086139 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:19.086142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:19.086235 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:19.086337 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:19.087057 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.087876 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.089140 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.089608 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.089740 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:19.093532 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:51461: BlockingRecv error: recv got EOF from 127.17.133.195:51461 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 14:53:19 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:19 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:19.203548 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:19.195407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:42081)
0925 14:53:19.195946 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:19.195962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:19.196074 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:19.196356 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:19.196359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:19.196800 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:19.196959 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:19.196968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.198185 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.198190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:19.199382 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:19.199394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.199994 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.199998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:19.200107 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:19.200718 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:19.200734 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:19.203321 (+ 2587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":297,"threads_started":1}
W20260925 14:53:19.203708 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:19.195478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:42081 (local address 127.17.133.254:45591)
0925 14:53:19.195700 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:19.195704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:19.196032 (+ 328us) server_negotiation.cc:407] Connection header received
0925 14:53:19.196167 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:19.196171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:19.196231 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:19.196314 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:19.197082 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.198040 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.199499 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.199874 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.199964 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:19.203483 (+ 3519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:42081: BlockingRecv error: recv got EOF from 127.17.133.193:42081 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 14:53:19 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:19 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:19.663067 27191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:19.655374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:36779)
0925 14:53:19.655891 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:19.655907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:19.656012 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:19.656306 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:19.656310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:19.656749 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:19.656910 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:19.656918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.657946 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.657950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:19.658897 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:19.658909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.659541 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.659545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:19.659659 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:19.660213 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:19.660229 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:19.662840 (+ 2611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":286,"threads_started":1}
W20260925 14:53:19.663208 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:19.655440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:36779 (local address 127.17.133.254:45591)
0925 14:53:19.655602 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:19.655606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:19.655978 (+ 372us) server_negotiation.cc:407] Connection header received
0925 14:53:19.656103 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:19.656106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:19.656170 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:19.656251 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:19.657025 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.657728 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.659016 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.659419 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.659507 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:19.662999 (+ 3492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:36779: BlockingRecv error: recv got EOF from 127.17.133.194:36779 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 14:53:20 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:20 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:20.102633 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:20.094453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:36887)
0925 14:53:20.094975 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:20.094995 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:20.095103 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:20.095383 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:20.095386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:20.095752 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:20.095903 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:20.095910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.096870 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.096874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:20.097790 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:20.097800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.098406 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.098409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:20.098521 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:20.099106 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:20.099131 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:20.102353 (+ 3222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":284,"threads_started":1}
W20260925 14:53:20.102743 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:20.094511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:36887 (local address 127.17.133.254:45591)
0925 14:53:20.094672 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:20.094676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:20.095069 (+ 393us) server_negotiation.cc:407] Connection header received
0925 14:53:20.095198 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:20.095202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:20.095258 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:20.095336 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:20.096030 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.096761 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.097905 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.098291 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.098379 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:20.102522 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:36887: BlockingRecv error: recv got EOF from 127.17.133.195:36887 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 14:53:20 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:20 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:20.218176 27193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:20.204676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:54853)
0925 14:53:20.205152 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:20.205167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:20.205277 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:20.205637 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:20.205640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:20.206118 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:20.206306 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:20.206319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.207461 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.207465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:20.208360 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:20.208370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.209037 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.209041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:20.209149 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:20.214768 (+ 5619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:20.214784 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:20.217800 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":281,"threads_started":1}
W20260925 14:53:20.218276 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:20.204758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:54853 (local address 127.17.133.254:45591)
0925 14:53:20.204943 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:20.204947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:20.205239 (+ 292us) server_negotiation.cc:407] Connection header received
0925 14:53:20.205387 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:20.205392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:20.205482 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:20.205586 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:20.206486 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.207340 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.208467 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.208898 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.209010 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:20.218012 (+ 9002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:54853: BlockingRecv error: recv got EOF from 127.17.133.193:54853 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 14:53:20 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:20 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:20.676060 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:20.664117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:44701)
0925 14:53:20.664689 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:20.664716 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:20.664865 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:20.665223 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:20.665228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:20.665748 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:20.665940 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:20.665958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.666903 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.666907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:20.667802 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:20.667812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.668419 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.668423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:20.668527 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:20.673074 (+ 4547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:20.673091 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:20.675735 (+ 2644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":304,"threads_started":1}
W20260925 14:53:20.676172 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:20.664242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:44701 (local address 127.17.133.254:45591)
0925 14:53:20.664470 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:20.664476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:20.664793 (+ 317us) server_negotiation.cc:407] Connection header received
0925 14:53:20.665000 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:20.665012 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:20.665072 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:20.665151 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:20.666075 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.666794 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.667916 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:20.668303 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:20.668387 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:20.675928 (+ 7541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:44701: BlockingRecv error: recv got EOF from 127.17.133.194:44701 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 14:53:21 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:21 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:21.111385 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:21.103560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:59795)
0925 14:53:21.104058 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:21.104080 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:21.104184 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:21.104452 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:21.104455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:21.104953 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:21.105093 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:21.105101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.106130 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.106134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:21.107049 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:21.107060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.107674 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.107678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:21.107783 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:21.108398 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:21.108415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:21.111031 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":277,"threads_started":1}
W20260925 14:53:21.111455 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:21.103707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:59795 (local address 127.17.133.254:45591)
0925 14:53:21.103912 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:21.103918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:21.104157 (+ 239us) server_negotiation.cc:407] Connection header received
0925 14:53:21.104263 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:21.104266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:21.104326 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:21.104403 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:21.105211 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.106030 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.107166 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.107554 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.107642 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:21.111229 (+ 3587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:59795: BlockingRecv error: recv got EOF from 127.17.133.195:59795 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 25 14:53:21 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:21 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:21.226843 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:21.219140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:34311)
0925 14:53:21.219633 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:21.219647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:21.219751 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:21.220030 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:21.220033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:21.220397 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:21.220563 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:21.220570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.221652 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.221656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:21.222603 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:21.222614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.223188 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.223192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:21.223301 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:21.223846 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:21.223862 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:21.226603 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":290,"threads_started":1}
W20260925 14:53:21.227041 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:21.219236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:34311 (local address 127.17.133.254:45591)
0925 14:53:21.219405 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:21.219409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:21.219715 (+ 306us) server_negotiation.cc:407] Connection header received
0925 14:53:21.219854 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:21.219857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:21.219913 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:21.219987 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:21.220746 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.221533 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.222708 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.223078 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.223160 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:21.226773 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:34311: BlockingRecv error: recv got EOF from 127.17.133.193:34311 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 14:53:21 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:21 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:21.685309 27197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:21.677108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:59681)
0925 14:53:21.677574 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:21.677589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:21.677745 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:21.678069 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:21.678074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:21.678533 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:21.678704 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:21.678713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.679778 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.679786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:21.680899 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:21.680911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.681608 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.681614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:21.681793 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:21.682436 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:21.682454 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:21.685087 (+ 2633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":242,"threads_started":1}
W20260925 14:53:21.685487 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:21.677179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:59681 (local address 127.17.133.254:45591)
0925 14:53:21.677393 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:21.677401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:21.677675 (+ 274us) server_negotiation.cc:407] Connection header received
0925 14:53:21.677859 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:21.677864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:21.677937 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:21.678042 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:21.678839 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.679652 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.681043 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.681461 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.681595 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:21.685252 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:59681: BlockingRecv error: recv got EOF from 127.17.133.194:59681 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 14:53:22 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:22 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:22.120527 27198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.112512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:42239)
0925 14:53:22.112974 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.112994 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.113107 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.113374 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.113377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.113729 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:22.113885 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.113892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.114849 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.114855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.115723 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.115734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.116415 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.116421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.116659 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.117390 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:22.117411 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:22.120297 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":273,"threads_started":1}
W20260925 14:53:22.120690 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:22.112573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:42239 (local address 127.17.133.254:45591)
0925 14:53:22.112790 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.112799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.113082 (+ 283us) server_negotiation.cc:407] Connection header received
0925 14:53:22.113199 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.113202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.113258 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.113335 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:22.114020 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.114714 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.115866 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.116269 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.116400 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.120463 (+ 4063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:42239: BlockingRecv error: recv got EOF from 127.17.133.195:42239 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 14:53:22 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:22 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:22.235371 27199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.227691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:51675)
0925 14:53:22.228198 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.228214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.228348 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.228682 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.228693 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.229083 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:22.229228 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.229236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.230243 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.230247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.231167 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.231177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.231831 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.231835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.231957 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.232534 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:22.232550 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:22.235155 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":295,"threads_started":1}
W20260925 14:53:22.235608 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:22.227764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:51675 (local address 127.17.133.254:45591)
0925 14:53:22.228008 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.228014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.228290 (+ 276us) server_negotiation.cc:407] Connection header received
0925 14:53:22.228469 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.228473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.228531 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.228659 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:22.229378 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.230129 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.231289 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.231719 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.231806 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.235333 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:51675: BlockingRecv error: recv got EOF from 127.17.133.193:51675 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 14:53:22 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:22 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:22.693948 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.686230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:57491)
0925 14:53:22.686766 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.686788 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.686963 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.687337 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.687341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.687715 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:22.687862 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.687870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.688936 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.688940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.689798 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.689808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.690395 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.690399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.690517 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.691086 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:22.691103 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:22.693725 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":308,"threads_started":1}
W20260925 14:53:22.694103 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:22.686304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:57491 (local address 127.17.133.254:45591)
0925 14:53:22.686528 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.686538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.686905 (+ 367us) server_negotiation.cc:407] Connection header received
0925 14:53:22.687105 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.687109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.687167 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.687282 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:22.687975 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.688799 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.689905 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.690287 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.690371 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.693909 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:57491: BlockingRecv error: recv got EOF from 127.17.133.194:57491 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 14:53:23 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:23 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:23.129580 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:23.121560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:51017)
0925 14:53:23.122109 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:23.122141 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:23.122482 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:23.122864 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:23.122869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:23.123254 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:23.123393 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:23.123401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.124303 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.124306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:23.125242 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:23.125253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.125900 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.125909 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:23.126052 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:23.126591 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:23.126607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:23.129375 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":310,"threads_started":1}
W20260925 14:53:23.129750 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:23.121582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:51017 (local address 127.17.133.254:45591)
0925 14:53:23.121751 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:23.121759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:23.122236 (+ 477us) server_negotiation.cc:407] Connection header received
0925 14:53:23.122642 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:23.122647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:23.122719 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:23.122797 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:23.123504 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.124189 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.125370 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.125787 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.125877 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:23.129534 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:51017: BlockingRecv error: recv got EOF from 127.17.133.195:51017 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 14:53:23 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:23 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:23.244117 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:23.236271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:42775)
0925 14:53:23.236794 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:23.236808 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:23.236908 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:23.237200 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:23.237203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:23.237545 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:23.237691 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:23.237699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.238680 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.238687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:23.239784 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:23.239798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.240532 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.240538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:23.240718 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:23.241243 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:23.241259 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:23.243782 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":263,"threads_started":1}
W20260925 14:53:23.244200 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:23.236466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:42775 (local address 127.17.133.254:45591)
0925 14:53:23.236702 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:23.236706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:23.236874 (+ 168us) server_negotiation.cc:407] Connection header received
0925 14:53:23.236995 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:23.236998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:23.237052 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:23.237162 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:23.237816 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.238547 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.239930 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.240352 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.240475 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:23.243957 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:42775: BlockingRecv error: recv got EOF from 127.17.133.193:42775 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 14:53:23 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:23 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:23.703397 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:23.695021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:51693)
0925 14:53:23.695521 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:23.695536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:23.695685 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:23.696033 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:23.696038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:23.696423 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:23.696574 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:23.696582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.697575 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.697581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:23.698585 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:23.698599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.699417 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.699424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:23.699586 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:23.700248 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:23.700277 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:23.703052 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":271,"threads_started":1}
W20260925 14:53:23.703452 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:23.695204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:51693 (local address 127.17.133.254:45591)
0925 14:53:23.695409 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:23.695416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:23.695616 (+ 200us) server_negotiation.cc:407] Connection header received
0925 14:53:23.695812 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:23.695821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:23.695895 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:23.696007 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:23.696722 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.697440 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.698704 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.699234 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.699384 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:23.703230 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:51693: BlockingRecv error: recv got EOF from 127.17.133.194:51693 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 14:53:24 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:24 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:24.138617 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:24.130504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:47813)
0925 14:53:24.130996 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:24.131013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:24.131155 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:24.131470 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:24.131474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:24.131851 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:24.131995 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:24.132003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.132998 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.133005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:24.133910 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:24.133922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.134628 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.134635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:24.134797 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:24.135333 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:24.135349 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:24.138276 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":262,"threads_started":1}
W20260925 14:53:24.138705 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:24.130567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:47813 (local address 127.17.133.254:45591)
0925 14:53:24.130767 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:24.130776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:24.131094 (+ 318us) server_negotiation.cc:407] Connection header received
0925 14:53:24.131270 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:24.131277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:24.131338 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:24.131447 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:24.132130 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.132867 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.134048 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.134459 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.134595 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:24.138470 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:47813: BlockingRecv error: recv got EOF from 127.17.133.195:47813 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 14:53:24 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:24 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:24.253224 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:24.245113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:41333)
0925 14:53:24.245569 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:24.245586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:24.245758 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:24.246074 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:24.246079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:24.246505 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:24.246669 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:24.246677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.247703 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.247709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:24.248780 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:24.248792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.249485 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.249491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:24.249658 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:24.250227 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:24.250244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:24.252904 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
W20260925 14:53:24.253334 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:24.245169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:41333 (local address 127.17.133.254:45591)
0925 14:53:24.245456 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:24.245462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:24.245672 (+ 210us) server_negotiation.cc:407] Connection header received
0925 14:53:24.245869 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:24.245874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:24.245949 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:24.246043 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:24.246801 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.247570 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.248911 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.249340 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.249467 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:24.253097 (+ 3630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:41333: BlockingRecv error: recv got EOF from 127.17.133.193:41333 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 25 14:53:24 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:24 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:24.712071 27206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:24.704345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:56245)
0925 14:53:24.704864 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:24.704880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:24.705074 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:24.705361 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:24.705364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:24.705691 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:24.705913 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:24.705928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.706900 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.706904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:24.707805 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:24.707823 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.708504 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.708508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:24.708700 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:24.709244 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:24.709260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:24.711838 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":257,"threads_started":1}
W20260925 14:53:24.712204 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:24.704417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:56245 (local address 127.17.133.254:45591)
0925 14:53:24.704620 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:24.704627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:24.704967 (+ 340us) server_negotiation.cc:407] Connection header received
0925 14:53:24.705183 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:24.705187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:24.705245 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:24.705320 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:24.706069 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.706788 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.707952 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:24.708382 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:24.708483 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:24.711997 (+ 3514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:56245: BlockingRecv error: recv got EOF from 127.17.133.194:56245 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 14:53:25 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:25 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.195@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:25.147835 27207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.139710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.195:55423)
0925 14:53:25.140225 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.140250 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:25.140378 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:25.140755 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:25.140759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:25.141161 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:25.141320 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.141329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.142313 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.142320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.143239 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.143249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.143944 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.143949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.144125 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.144745 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:25.144761 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:25.147480 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":252,"threads_started":1}
W20260925 14:53:25.147917 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:25.139757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:55423 (local address 127.17.133.254:45591)
0925 14:53:25.139963 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:25.139971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:25.140312 (+ 341us) server_negotiation.cc:407] Connection header received
0925 14:53:25.140505 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:25.140510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:25.140627 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:25.140731 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:25.141462 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.142167 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.143372 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.143791 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.143934 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.147692 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.195:55423: BlockingRecv error: recv got EOF from 127.17.133.195:55423 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 14:53:25 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:25 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.193@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:25.262212 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.254284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.193:50907)
0925 14:53:25.254760 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.254776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:25.254886 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:25.255162 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:25.255165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:25.255583 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:25.255807 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.255816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.257002 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.257005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.257902 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.257912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.258524 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.258527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.258646 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.259290 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:25.259305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:25.261861 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":276,"threads_started":1}
W20260925 14:53:25.262267 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:25.254447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:50907 (local address 127.17.133.254:45591)
0925 14:53:25.254661 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:25.254667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:25.254843 (+ 176us) server_negotiation.cc:407] Connection header received
0925 14:53:25.254973 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:25.254976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:25.255047 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:25.255124 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:25.255960 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.256902 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.258007 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.258407 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.258491 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.262038 (+ 3547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.193:50907: BlockingRecv error: recv got EOF from 127.17.133.193:50907 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 14:53:25 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:53:25 dist-test-slave-v5vl krb5kdc[26164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.133.194@KRBTEST.COM for oryx/127.17.133.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:53:25.725875 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.713222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45591 (local address 127.17.133.194:50725)
0925 14:53:25.713884 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.713906 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:25.714038 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:25.714379 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:25.714384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:25.714785 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:25.714934 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.714941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.715937 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.715944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.716936 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.716947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.717656 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.717663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.717827 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.722817 (+ 4990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:25.722834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:25.725569 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.133.254:45591: Server oryx/127.17.133.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":403,"threads_started":1}
W20260925 14:53:25.726068 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:25.713277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:50725 (local address 127.17.133.254:45591)
0925 14:53:25.713425 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:25.713429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:25.713980 (+ 551us) server_negotiation.cc:407] Connection header received
0925 14:53:25.714176 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:25.714181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:25.714250 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:25.714352 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:25.715073 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.715800 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.717067 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.717497 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.717645 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.725802 (+ 8157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.133.194:50725: BlockingRecv error: recv got EOF from 127.17.133.194:50725 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:53:25.994369 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 26751
I20260925 14:53:26.005707 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.995304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.17.133.193:37921)
0925 14:53:25.995820 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:25.995827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:25.995852 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:53:25.995930 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:25.995937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:25.996152 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:25.996314 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:25.997700 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.998726 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.999837 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.000366 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.000458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.002693 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.002700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.002713 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.002744 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.005182 (+ 2438us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.133.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 14:53:26.005488 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57558: Request ticket server kudu/127.17.133.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":257,"threads_started":1}
W20260925 14:53:26.005818 27211 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57558: Request ticket server kudu/127.17.133.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 14:53:26.005846 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 26751: Not authorized: Client connection negotiation failed: client connection to 127.17.133.193:37921: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.133.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 14:53:26.006029 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 26751
I20260925 14:53:26.022008 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 26902
I20260925 14:53:26.033864 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.023126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.17.133.194:35157)
0925 14:53:26.023649 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.023656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.023684 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:53:26.023739 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.023745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.023953 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.024140 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.025404 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.026409 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.027862 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.028423 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.028553 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.030996 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.031005 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.031018 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.031051 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.033220 (+ 2169us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.133.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 14:53:26.033524 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39736: Request ticket server kudu/127.17.133.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
W20260925 14:53:26.033861 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 26902: Not authorized: Client connection negotiation failed: client connection to 127.17.133.194:35157: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.133.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 14:53:26.033975 27212 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39736: Request ticket server kudu/127.17.133.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 14:53:26.034034 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 26902
I20260925 14:53:26.050218 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 27033
I20260925 14:53:26.060847 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.051022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.17.133.195:40815)
0925 14:53:26.051619 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.051627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.051651 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:53:26.051712 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.051718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.051932 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.052076 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.053274 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.054246 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.055271 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.055826 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.055913 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.058140 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.058148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.058160 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.058187 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.060286 (+ 2099us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.133.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 14:53:26.060495 (+ 209us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47464: Request ticket server kudu/127.17.133.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":334,"threads_started":1}
W20260925 14:53:26.060891 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 27033: Not authorized: Client connection negotiation failed: client connection to 127.17.133.195:40815: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.133.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 14:53:26.060940 27213 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47464: Request ticket server kudu/127.17.133.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 14:53:26.061017 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 27033
I20260925 14:53:26.076789 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 26682
I20260925 14:53:26.078002 26744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:26.191587 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 26682
2026-09-25T14:53:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20887 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T14:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:26Z Disabled control of system clock
I20260925 14:53:26.261713 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:33535
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:33535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:53:26.573830 27218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:26.574115 27218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:26.574191 27218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:26.582064 27218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:26.582197 27218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:26.582266 27218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:26.582345 27218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:26.582412 27218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:26.592209 27218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:33535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:33535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:26.593492 27218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:26.595120 27218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:26.607043 27223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:26.606963 27226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:26.606997 27224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:26.607623 27218 server_base.cc:1034] running on GCE node
I20260925 14:53:26.608446 27218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:26.609884 27218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:26.611102 27218 hybrid_clock.cc:648] HybridClock initialized: now 1790348006611056 us; error 60 us; skew 500 ppm
I20260925 14:53:26.611431 27218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:53:26.613626 27218 webserver.cc:460] Webserver started at http://127.17.133.254:36689/ using document root <none> and password file <none>
I20260925 14:53:26.614285 27218 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:26.614413 27218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:26.614769 27218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:26.617724 27218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "ac6ea553ef834d229e3a672d92c23a57"
format_stamp: "Formatted at 2026-09-25 14:53:26 on dist-test-slave-v5vl"
server_key: "b64608a2ab80c2f5d560fe371d213a16"
server_key_iv: "6708728ec3be59b839fc35d9f3587905"
server_key_version: "encryptionkey@0"
I20260925 14:53:26.618495 27218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "ac6ea553ef834d229e3a672d92c23a57"
format_stamp: "Formatted at 2026-09-25 14:53:26 on dist-test-slave-v5vl"
server_key: "b64608a2ab80c2f5d560fe371d213a16"
server_key_iv: "6708728ec3be59b839fc35d9f3587905"
server_key_version: "encryptionkey@0"
I20260925 14:53:26.623469 27218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:53:26.626844 27232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:26.628145 27218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:53:26.628319 27218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "ac6ea553ef834d229e3a672d92c23a57"
format_stamp: "Formatted at 2026-09-25 14:53:26 on dist-test-slave-v5vl"
server_key: "b64608a2ab80c2f5d560fe371d213a16"
server_key_iv: "6708728ec3be59b839fc35d9f3587905"
server_key_version: "encryptionkey@0"
I20260925 14:53:26.628531 27218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:26.638809 27218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:26.639596 27218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:26.639880 27218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:26.657222 27218 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:33535
I20260925 14:53:26.657240 27283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:33535 every 8 connection(s)
I20260925 14:53:26.658532 27218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:26.662036 27284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:26.668785 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 27218
I20260925 14:53:26.668995 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:53:26.669812 17943 external_mini_cluster.cc:1428] Setting key 9c6c228881aae8dfff4ad41d370b103c
I20260925 14:53:26.673113 27284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57: Bootstrap starting.
I20260925 14:53:26.676031 27284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:26.677122 27284 log.cc:824] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:26.679654 27284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57: No bootstrap required, opened a new log
I20260925 14:53:26.682919 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.671860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.17.133.254:33535)
0925 14:53:26.672840 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.672852 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.672893 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:53:26.674533 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.674554 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.674980 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.675344 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.675979 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.677383 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.678669 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.679304 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.679494 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.680726 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.680733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.680749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:26.681248 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.681320 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:26.681341 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.681504 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.681719 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.681987 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":312,"threads_started":1}
I20260925 14:53:26.687371 27284 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ea553ef834d229e3a672d92c23a57" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33535 } }
I20260925 14:53:26.687726 27284 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:26.687839 27284 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6ea553ef834d229e3a672d92c23a57, State: Initialized, Role: FOLLOWER
I20260925 14:53:26.688405 27284 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ea553ef834d229e3a672d92c23a57" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33535 } }
I20260925 14:53:26.688666 27284 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:26.688823 27284 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:26.689003 27284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:26.691449 27284 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ea553ef834d229e3a672d92c23a57" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33535 } }
I20260925 14:53:26.691900 27284 leader_election.cc:302] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac6ea553ef834d229e3a672d92c23a57; no voters:
I20260925 14:53:26.692461 27284 leader_election.cc:288] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:26.692613 27289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:26.693313 27289 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [term 1 LEADER]: Becoming Leader. State: Replica: ac6ea553ef834d229e3a672d92c23a57, State: Running, Role: LEADER
I20260925 14:53:26.693796 27289 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ea553ef834d229e3a672d92c23a57" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33535 } }
I20260925 14:53:26.694412 27284 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:26.696496 27291 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac6ea553ef834d229e3a672d92c23a57. Latest consensus state: current_term: 1 leader_uuid: "ac6ea553ef834d229e3a672d92c23a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ea553ef834d229e3a672d92c23a57" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33535 } } }
I20260925 14:53:26.697134 27291 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:26.698365 27290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac6ea553ef834d229e3a672d92c23a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ea553ef834d229e3a672d92c23a57" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33535 } } }
I20260925 14:53:26.698783 27290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:26.699177 27296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:26.704154 27296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:26.711706 27296 catalog_manager.cc:1347] Generated new cluster ID: f7a26ebead294acda1082b20b3aaf446
I20260925 14:53:26.711809 27296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:26.719007 27296 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:26.720731 27296 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:26.729921 27296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac6ea553ef834d229e3a672d92c23a57: Generated new TSK 0
I20260925 14:53:26.730610 27296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:26.741815 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33535
--builtin_ntp_servers=127.17.133.212:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:53:27.056113 27308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:27.056398 27308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:27.056500 27308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:27.063712 27308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:27.063807 27308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:27.063942 27308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:53:27.072891 27308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:27.074143 27308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:27.075861 27308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:27.088284 27313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:27.088320 27314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:27.088284 27316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:27.089504 27308 server_base.cc:1034] running on GCE node
I20260925 14:53:27.089973 27308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:27.090773 27308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:27.091971 27308 hybrid_clock.cc:648] HybridClock initialized: now 1790348007091940 us; error 27 us; skew 500 ppm
I20260925 14:53:27.092263 27308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:53:27.094365 27308 webserver.cc:460] Webserver started at http://127.17.133.193:41963/ using document root <none> and password file <none>
I20260925 14:53:27.095039 27308 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:27.095160 27308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:27.095484 27308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:27.098116 27308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "1aa84e62b588497e813a52ad08237f5b"
format_stamp: "Formatted at 2026-09-25 14:53:27 on dist-test-slave-v5vl"
server_key: "af136b741cfcd9e9828d431be57ae732"
server_key_iv: "fd9cfe408355d8f34709737dc1f2f4f5"
server_key_version: "encryptionkey@0"
I20260925 14:53:27.098867 27308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "1aa84e62b588497e813a52ad08237f5b"
format_stamp: "Formatted at 2026-09-25 14:53:27 on dist-test-slave-v5vl"
server_key: "af136b741cfcd9e9828d431be57ae732"
server_key_iv: "fd9cfe408355d8f34709737dc1f2f4f5"
server_key_version: "encryptionkey@0"
I20260925 14:53:27.103535 27308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 14:53:27.106653 27322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:27.107858 27308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 14:53:27.108009 27308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "1aa84e62b588497e813a52ad08237f5b"
format_stamp: "Formatted at 2026-09-25 14:53:27 on dist-test-slave-v5vl"
server_key: "af136b741cfcd9e9828d431be57ae732"
server_key_iv: "fd9cfe408355d8f34709737dc1f2f4f5"
server_key_version: "encryptionkey@0"
I20260925 14:53:27.108210 27308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:27.123675 27308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:27.124439 27308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:27.124776 27308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:27.125633 27308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:27.126899 27308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:27.127007 27308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:27.127125 27308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:27.127219 27308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:27.158344 27308 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:36929
I20260925 14:53:27.158440 27434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:36929 every 8 connection(s)
I20260925 14:53:27.159569 27308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:27.161880 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 27308
I20260925 14:53:27.162088 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:27.162864 17943 external_mini_cluster.cc:1428] Setting key 8539415e36d6f3c3a8a76931cf50cd18
I20260925 14:53:27.165505 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33535
--builtin_ntp_servers=127.17.133.212:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 14:53:27.171998 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:27.162120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:47727 (local address 127.17.133.254:33535)
0925 14:53:27.162312 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:27.162316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:27.163902 (+ 1586us) server_negotiation.cc:407] Connection header received
0925 14:53:27.165738 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:27.165742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:27.165802 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:27.165854 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:27.167103 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.167812 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.169474 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.169833 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.169954 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:27.171325 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:27.171330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:27.171333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:27.171349 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:27.171377 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:27.171389 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:27.171459 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:27.171688 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 14:53:27.171751 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:53:27.173384 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:27.162780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33535 (local address 127.17.133.193:47727)
0925 14:53:27.163772 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:27.163806 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:27.165445 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:27.166225 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:27.166232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:27.166318 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:27.166921 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:27.166955 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.167948 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.167954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:27.169328 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:27.169339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.170026 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.170032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:27.170233 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:27.170979 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:27.170983 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:27.171035 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:27.171129 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:27.171479 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:27.171491 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:27.171499 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:53:27.171679 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 14:53:27.172100 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":295,"threads_started":1}
I20260925 14:53:27.174738 27435 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33535
I20260925 14:53:27.175001 27435 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:27.175710 27435 heartbeater.cc:511] Master 127.17.133.254:33535 requested a full tablet report, sending...
I20260925 14:53:27.177578 27249 ts_manager.cc:194] Registered new tserver with Master: 1aa84e62b588497e813a52ad08237f5b (127.17.133.193:36929)
I20260925 14:53:27.179059 27249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.193:47727
W20260925 14:53:27.474678 27438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:27.474987 27438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:27.475143 27438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:27.481987 27438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:27.482113 27438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:27.482285 27438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:53:27.491158 27438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:27.492368 27438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:27.494091 27438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:27.506675 27443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:27.506709 27444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:27.506569 27446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:27.507550 27438 server_base.cc:1034] running on GCE node
I20260925 14:53:27.508118 27438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:27.509002 27438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:27.510211 27438 hybrid_clock.cc:648] HybridClock initialized: now 1790348007510181 us; error 53 us; skew 500 ppm
I20260925 14:53:27.510493 27438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:53:27.512728 27438 webserver.cc:460] Webserver started at http://127.17.133.194:45163/ using document root <none> and password file <none>
I20260925 14:53:27.513353 27438 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:27.513478 27438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:27.513782 27438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:27.516309 27438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "8ffc83869bcc478e838e700091befe2f"
format_stamp: "Formatted at 2026-09-25 14:53:27 on dist-test-slave-v5vl"
server_key: "96d3624b0362f52caffea48f3f79fb76"
server_key_iv: "5765305c17317172c6181d4b4d2c38ca"
server_key_version: "encryptionkey@0"
I20260925 14:53:27.517084 27438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "8ffc83869bcc478e838e700091befe2f"
format_stamp: "Formatted at 2026-09-25 14:53:27 on dist-test-slave-v5vl"
server_key: "96d3624b0362f52caffea48f3f79fb76"
server_key_iv: "5765305c17317172c6181d4b4d2c38ca"
server_key_version: "encryptionkey@0"
I20260925 14:53:27.521634 27438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 14:53:27.524653 27452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:27.525785 27438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 14:53:27.525962 27438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "8ffc83869bcc478e838e700091befe2f"
format_stamp: "Formatted at 2026-09-25 14:53:27 on dist-test-slave-v5vl"
server_key: "96d3624b0362f52caffea48f3f79fb76"
server_key_iv: "5765305c17317172c6181d4b4d2c38ca"
server_key_version: "encryptionkey@0"
I20260925 14:53:27.526155 27438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:27.539893 27438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:27.540576 27438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:27.540915 27438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:27.541760 27438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:27.543012 27438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:27.543130 27438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:27.543262 27438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:27.543350 27438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:27.574003 27438 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:38777
I20260925 14:53:27.574050 27564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:38777 every 8 connection(s)
I20260925 14:53:27.575281 27438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:27.577884 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 27438
I20260925 14:53:27.578085 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:27.578631 17943 external_mini_cluster.cc:1428] Setting key bcf948612948df0685d48ea51553d15c
I20260925 14:53:27.581224 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33535
--builtin_ntp_servers=127.17.133.212:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 14:53:27.588719 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:27.577638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:49157 (local address 127.17.133.254:33535)
0925 14:53:27.577794 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:27.577798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:27.579515 (+ 1717us) server_negotiation.cc:407] Connection header received
0925 14:53:27.581297 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:27.581302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:27.581410 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:27.581520 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:27.582520 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.583211 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.585564 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.586168 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.586260 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:27.587899 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:27.587907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:27.587916 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:27.587932 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:27.587962 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:27.587973 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:27.588067 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:27.588288 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:53:27.588374 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:53:27.589699 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:27.578337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33535 (local address 127.17.133.194:49157)
0925 14:53:27.579341 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:27.579398 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:27.581055 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:27.581736 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:27.581743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:27.581824 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:27.582357 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:27.582372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.583376 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.583380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:27.585384 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:27.585401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.586574 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.586581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:27.586875 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:27.587600 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:27.587603 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:27.587639 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:27.587721 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:27.588066 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:27.588077 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:27.588087 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:53:27.588268 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 14:53:27.588655 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":308,"threads_started":1}
I20260925 14:53:27.591084 27565 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33535
I20260925 14:53:27.591370 27565 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:27.592060 27565 heartbeater.cc:511] Master 127.17.133.254:33535 requested a full tablet report, sending...
I20260925 14:53:27.593444 27249 ts_manager.cc:194] Registered new tserver with Master: 8ffc83869bcc478e838e700091befe2f (127.17.133.194:38777)
I20260925 14:53:27.594609 27249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.194:49157
W20260925 14:53:27.900120 27568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:27.900362 27568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:27.900524 27568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:27.907707 27568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:27.907809 27568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:27.907966 27568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:53:27.917128 27568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:27.918404 27568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:27.920132 27568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:27.933476 27573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:27.933414 27574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:27.933671 27576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:27.934090 27568 server_base.cc:1034] running on GCE node
I20260925 14:53:27.934719 27568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:27.935642 27568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:27.936860 27568 hybrid_clock.cc:648] HybridClock initialized: now 1790348007936810 us; error 41 us; skew 500 ppm
I20260925 14:53:27.937177 27568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:53:27.939281 27568 webserver.cc:460] Webserver started at http://127.17.133.195:38111/ using document root <none> and password file <none>
I20260925 14:53:27.939958 27568 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:27.940081 27568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:27.940388 27568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:27.943118 27568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "79ef4b795abe4a54b606c0e35fda1812"
format_stamp: "Formatted at 2026-09-25 14:53:27 on dist-test-slave-v5vl"
server_key: "aa512f1515f8ba010526bb4be926c890"
server_key_iv: "236c052e444aefbb610bf44a989322b6"
server_key_version: "encryptionkey@0"
I20260925 14:53:27.943971 27568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "79ef4b795abe4a54b606c0e35fda1812"
format_stamp: "Formatted at 2026-09-25 14:53:27 on dist-test-slave-v5vl"
server_key: "aa512f1515f8ba010526bb4be926c890"
server_key_iv: "236c052e444aefbb610bf44a989322b6"
server_key_version: "encryptionkey@0"
I20260925 14:53:27.948678 27568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 14:53:27.951716 27582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:27.952943 27568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:53:27.953106 27568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "79ef4b795abe4a54b606c0e35fda1812"
format_stamp: "Formatted at 2026-09-25 14:53:27 on dist-test-slave-v5vl"
server_key: "aa512f1515f8ba010526bb4be926c890"
server_key_iv: "236c052e444aefbb610bf44a989322b6"
server_key_version: "encryptionkey@0"
I20260925 14:53:27.953301 27568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:27.979485 27568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:27.980316 27568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:27.980662 27568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:27.981595 27568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:27.983017 27568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:27.983136 27568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:27.983259 27568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:27.983351 27568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:28.015187 27568 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:33459
I20260925 14:53:28.015249 27694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:33459 every 8 connection(s)
I20260925 14:53:28.016525 27568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:28.025125 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 27568
I20260925 14:53:28.025302 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:28.025785 17943 external_mini_cluster.cc:1428] Setting key 807b053f3fd2902b2f0c9161c30ce2ba
I20260925 14:53:28.028872 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.019016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:52991 (local address 127.17.133.254:33535)
0925 14:53:28.019187 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.019192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.020545 (+ 1353us) server_negotiation.cc:407] Connection header received
0925 14:53:28.022225 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.022229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.022289 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.022373 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.023318 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.024006 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.025997 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.026493 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.026592 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.028094 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.028100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.028104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:28.028123 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.028158 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:28.028176 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.028266 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.028467 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.028574 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:53:28.029867 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.019545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33535 (local address 127.17.133.195:52991)
0925 14:53:28.020403 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.020440 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.021997 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.022551 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.022558 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.022639 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.023190 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.023204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.024162 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.024169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.025848 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.025861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.026696 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.026705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.026973 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.027789 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:28.027793 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.027844 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:28.027924 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.028270 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.028276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.028281 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.028418 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.028825 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":236,"threads_started":1}
I20260925 14:53:28.030975 27695 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33535
I20260925 14:53:28.031239 27695 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:28.032017 27695 heartbeater.cc:511] Master 127.17.133.254:33535 requested a full tablet report, sending...
I20260925 14:53:28.033272 27249 ts_manager.cc:194] Registered new tserver with Master: 79ef4b795abe4a54b606c0e35fda1812 (127.17.133.195:33459)
I20260925 14:53:28.034057 27249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.195:52991
I20260925 14:53:28.040694 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:53:28.046285 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:28.044820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.17.133.254:33535)
0925 14:53:28.045035 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.045044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.045444 (+ 400us) server_negotiation.cc:407] Connection header received
0925 14:53:28.045556 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.045559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.045612 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.045689 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.046066 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37280: BlockingRecv error: recv got EOF from 127.0.0.1:37280 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 14:53:28.046628 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 27308
I20260925 14:53:28.053880 27704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.047621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.17.133.193:36929)
0925 14:53:28.048141 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.048149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.048182 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:53:28.048240 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.048245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.048425 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.048544 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.048915 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.049835 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.050982 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.051493 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.051584 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.052574 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.052623 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.052639 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:28.053221 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.053266 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:28.053283 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.053366 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.053493 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.053599 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":299,"threads_started":1}
I20260925 14:53:28.054987 27430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:28.135963 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 27308
I20260925 14:53:28.151646 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 27438
I20260925 14:53:28.159070 27706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.152534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.17.133.194:38777)
0925 14:53:28.153144 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.153158 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.153186 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:53:28.153249 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.153255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.153469 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.153602 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.153933 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.154884 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.156035 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.156704 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.156793 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.157816 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.157823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.157837 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:28.158423 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.158469 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:28.158486 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.158570 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.158664 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.158774 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":365,"threads_started":1}
I20260925 14:53:28.160322 27560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:28.241015 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 27438
I20260925 14:53:28.256349 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 27568
I20260925 14:53:28.264253 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.257379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59624 (local address 127.17.133.195:33459)
0925 14:53:28.258013 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.258019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.258047 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:53:28.258104 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.258109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.258315 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.258481 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.258914 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.259913 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.261245 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.261772 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.261901 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.262947 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.262955 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.262967 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:28.263502 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.263545 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:28.263563 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.263694 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.263801 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.263918 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":358,"threads_started":1}
I20260925 14:53:28.265540 27690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:28.344473 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 27568
I20260925 14:53:28.361552 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 27218
I20260925 14:53:28.362712 27279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:28.459846 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 27218
2026-09-25T14:53:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2263 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T14:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:28Z Disabled control of system clock
I20260925 14:53:28.527482 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46631
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:46631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:53:28.834645 27714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:28.834910 27714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:28.834983 27714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:28.841686 27714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:28.841776 27714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:28.841861 27714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:28.841969 27714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:28.842051 27714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:28.850709 27714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:46631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:28.851881 27714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:28.853626 27714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:28.864506 27719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:28.864620 27720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:28.864560 27722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:28.865695 27714 server_base.cc:1034] running on GCE node
I20260925 14:53:28.866324 27714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:28.867452 27714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:28.868671 27714 hybrid_clock.cc:648] HybridClock initialized: now 1790348008868622 us; error 43 us; skew 500 ppm
I20260925 14:53:28.868990 27714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:53:28.871163 27714 webserver.cc:460] Webserver started at http://127.17.133.254:44623/ using document root <none> and password file <none>
I20260925 14:53:28.871836 27714 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:28.871942 27714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:28.872242 27714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:28.874918 27714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "7458933e606b4858b71006eabd93d702"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-v5vl"
server_key: "3c51dd5628b82319aa8c613b3cdeffdc"
server_key_iv: "6342525bab91a83903255e35d48807ca"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.875661 27714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "7458933e606b4858b71006eabd93d702"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-v5vl"
server_key: "3c51dd5628b82319aa8c613b3cdeffdc"
server_key_iv: "6342525bab91a83903255e35d48807ca"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.880100 27714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 14:53:28.883098 27728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:28.884238 27714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:28.884402 27714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "7458933e606b4858b71006eabd93d702"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-v5vl"
server_key: "3c51dd5628b82319aa8c613b3cdeffdc"
server_key_iv: "6342525bab91a83903255e35d48807ca"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.884608 27714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:28.889539 27714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:28.890203 27714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:28.890476 27714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:28.907348 27714 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:46631
I20260925 14:53:28.907385 27779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:46631 every 8 connection(s)
I20260925 14:53:28.908597 27714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:28.911953 27780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:28.913883 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 27714
I20260925 14:53:28.914072 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:53:28.914718 17943 external_mini_cluster.cc:1428] Setting key 167bf77c0292093380a64b1116f4d5f6
I20260925 14:53:28.922511 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.916572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.17.133.254:46631)
0925 14:53:28.917579 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.917595 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.917634 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:53:28.919203 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.919222 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.919700 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.920005 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.921089 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.921095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.921106 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:28.921122 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.921174 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:28.921188 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.921348 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.921478 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.921735 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":372,"threads_started":1}
I20260925 14:53:28.923115 27780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702: Bootstrap starting.
I20260925 14:53:28.926286 27780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:28.927158 27780 log.cc:824] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:28.929592 27780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702: No bootstrap required, opened a new log
I20260925 14:53:28.936474 27780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7458933e606b4858b71006eabd93d702" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46631 } }
I20260925 14:53:28.936913 27780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:28.937011 27780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7458933e606b4858b71006eabd93d702, State: Initialized, Role: FOLLOWER
I20260925 14:53:28.937618 27780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7458933e606b4858b71006eabd93d702" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46631 } }
I20260925 14:53:28.937845 27780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:28.937980 27780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:28.938175 27780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:28.940205 27780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7458933e606b4858b71006eabd93d702" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46631 } }
I20260925 14:53:28.940753 27780 leader_election.cc:302] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7458933e606b4858b71006eabd93d702; no voters:
I20260925 14:53:28.941375 27780 leader_election.cc:288] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:28.941501 27785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:28.942201 27785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [term 1 LEADER]: Becoming Leader. State: Replica: 7458933e606b4858b71006eabd93d702, State: Running, Role: LEADER
I20260925 14:53:28.942690 27785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7458933e606b4858b71006eabd93d702" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46631 } }
I20260925 14:53:28.943508 27780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:28.945554 27786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7458933e606b4858b71006eabd93d702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7458933e606b4858b71006eabd93d702" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46631 } } }
I20260925 14:53:28.945613 27787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7458933e606b4858b71006eabd93d702. Latest consensus state: current_term: 1 leader_uuid: "7458933e606b4858b71006eabd93d702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7458933e606b4858b71006eabd93d702" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46631 } } }
I20260925 14:53:28.945960 27786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:28.945957 27787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:28.947887 27792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:28.953078 27792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:28.961329 27792 catalog_manager.cc:1347] Generated new cluster ID: f05a925a0a584949a1dbc1894530aa92
I20260925 14:53:28.961457 27792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:28.977059 27792 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:28.977473 27792 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:28.984161 27792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7458933e606b4858b71006eabd93d702: Generated new TSK 0
I20260925 14:53:28.984949 27792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:28.992920 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46631
--builtin_ntp_servers=127.17.133.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:53:29.315838 27804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:29.316121 27804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:29.316210 27804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:29.323314 27804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:29.323405 27804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:29.323529 27804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:53:29.332322 27804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:29.333592 27804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:29.335291 27804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:29.348186 27809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:29.348141 27810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:29.348217 27812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:29.348992 27804 server_base.cc:1034] running on GCE node
I20260925 14:53:29.349606 27804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:29.350513 27804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:29.351737 27804 hybrid_clock.cc:648] HybridClock initialized: now 1790348009351701 us; error 50 us; skew 500 ppm
I20260925 14:53:29.352059 27804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:53:29.354447 27804 webserver.cc:460] Webserver started at http://127.17.133.193:37739/ using document root <none> and password file <none>
I20260925 14:53:29.355145 27804 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:29.355271 27804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:29.355579 27804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:29.359542 27804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "4838929d34c44333a88da32641ade3ea"
format_stamp: "Formatted at 2026-09-25 14:53:29 on dist-test-slave-v5vl"
server_key: "49fe8b9a23c4faabd394039ea46275f6"
server_key_iv: "6d68ccec40cd1bead89ae96c62169901"
server_key_version: "encryptionkey@0"
I20260925 14:53:29.360313 27804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "4838929d34c44333a88da32641ade3ea"
format_stamp: "Formatted at 2026-09-25 14:53:29 on dist-test-slave-v5vl"
server_key: "49fe8b9a23c4faabd394039ea46275f6"
server_key_iv: "6d68ccec40cd1bead89ae96c62169901"
server_key_version: "encryptionkey@0"
I20260925 14:53:29.365257 27804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260925 14:53:29.368389 27818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:29.369652 27804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:53:29.369817 27804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "4838929d34c44333a88da32641ade3ea"
format_stamp: "Formatted at 2026-09-25 14:53:29 on dist-test-slave-v5vl"
server_key: "49fe8b9a23c4faabd394039ea46275f6"
server_key_iv: "6d68ccec40cd1bead89ae96c62169901"
server_key_version: "encryptionkey@0"
I20260925 14:53:29.370024 27804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:29.375556 27804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:29.376303 27804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:29.376577 27804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:29.377516 27804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:29.378829 27804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:29.378952 27804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:29.379093 27804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:29.379185 27804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:29.410279 27804 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:44411
I20260925 14:53:29.410328 27930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:44411 every 8 connection(s)
I20260925 14:53:29.411509 27804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:29.416250 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 27804
I20260925 14:53:29.416450 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:29.417132 17943 external_mini_cluster.cc:1428] Setting key 63d4a1b009eed081f9be29b48e485fdc
I20260925 14:53:29.419669 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:29.414093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:57357 (local address 127.17.133.254:46631)
0925 14:53:29.414261 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:29.414265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:29.415620 (+ 1355us) server_negotiation.cc:407] Connection header received
0925 14:53:29.417445 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:29.417451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:29.417524 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:29.417636 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:29.418965 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:29.418970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:29.418975 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:29.418999 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:29.419027 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:29.419039 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:29.419150 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:29.419354 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 14:53:29.419418 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:53:29.419836 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46631
--builtin_ntp_servers=127.17.133.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 14:53:29.420727 27933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:29.414612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:46631 (local address 127.17.133.193:57357)
0925 14:53:29.415461 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:29.415496 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:29.417148 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:29.417866 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:29.417874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:29.417966 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:29.418657 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:29.418660 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:29.418711 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:29.418815 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:29.419137 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:29.419146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:29.419153 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:53:29.419425 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 14:53:29.419685 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":260,"threads_started":1}
I20260925 14:53:29.422163 27931 heartbeater.cc:347] Connected to a master server at 127.17.133.254:46631
I20260925 14:53:29.422473 27931 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:29.423084 27931 heartbeater.cc:511] Master 127.17.133.254:46631 requested a full tablet report, sending...
I20260925 14:53:29.425201 27745 ts_manager.cc:194] Registered new tserver with Master: 4838929d34c44333a88da32641ade3ea (127.17.133.193:44411)
W20260925 14:53:29.733577 27934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:29.733836 27934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:29.733994 27934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:29.741336 27934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:29.741430 27934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:29.741629 27934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:53:29.750991 27934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:29.752211 27934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:29.754034 27934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:29.766407 27939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:29.766424 27940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:29.766467 27942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:29.767195 27934 server_base.cc:1034] running on GCE node
I20260925 14:53:29.767738 27934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:29.768656 27934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:29.769871 27934 hybrid_clock.cc:648] HybridClock initialized: now 1790348009769827 us; error 39 us; skew 500 ppm
I20260925 14:53:29.770174 27934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:53:29.772284 27934 webserver.cc:460] Webserver started at http://127.17.133.194:39473/ using document root <none> and password file <none>
I20260925 14:53:29.773041 27934 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:29.773159 27934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:29.773511 27934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:29.776261 27934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "d087e0406c654b3c90c3129ad978b4d4"
format_stamp: "Formatted at 2026-09-25 14:53:29 on dist-test-slave-v5vl"
server_key: "429d3830e75b5a47b80293c5530decac"
server_key_iv: "7108ab3d60baa12f6a7d6df5367454ab"
server_key_version: "encryptionkey@0"
I20260925 14:53:29.777031 27934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "d087e0406c654b3c90c3129ad978b4d4"
format_stamp: "Formatted at 2026-09-25 14:53:29 on dist-test-slave-v5vl"
server_key: "429d3830e75b5a47b80293c5530decac"
server_key_iv: "7108ab3d60baa12f6a7d6df5367454ab"
server_key_version: "encryptionkey@0"
I20260925 14:53:29.782214 27934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:53:29.785363 27948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:29.786580 27934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:53:29.786746 27934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "d087e0406c654b3c90c3129ad978b4d4"
format_stamp: "Formatted at 2026-09-25 14:53:29 on dist-test-slave-v5vl"
server_key: "429d3830e75b5a47b80293c5530decac"
server_key_iv: "7108ab3d60baa12f6a7d6df5367454ab"
server_key_version: "encryptionkey@0"
I20260925 14:53:29.786957 27934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:29.792256 27934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:29.793023 27934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:29.793277 27934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:29.794121 27934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:29.795367 27934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:29.795485 27934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:29.795621 27934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:29.795717 27934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:29.826472 27934 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:44445
I20260925 14:53:29.826519 28060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:44445 every 8 connection(s)
I20260925 14:53:29.827783 27934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:29.831508 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 27934
I20260925 14:53:29.831662 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:29.832226 17943 external_mini_cluster.cc:1428] Setting key 68b7121acd71706d9228b9ef7927c686
I20260925 14:53:29.835112 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46631
--builtin_ntp_servers=127.17.133.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 14:53:29.836108 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:29.830230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:52653 (local address 127.17.133.254:46631)
0925 14:53:29.830399 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:29.830403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:29.831885 (+ 1482us) server_negotiation.cc:407] Connection header received
0925 14:53:29.833975 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:29.833980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:29.834035 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:29.834136 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:29.835515 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:29.835519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:29.835522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:29.835535 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:29.835562 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:29.835572 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:29.835657 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:29.835858 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 14:53:29.835900 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:53:29.837421 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:29.830673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:46631 (local address 127.17.133.194:52653)
0925 14:53:29.831705 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:29.831756 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:29.833689 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:29.834368 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:29.834378 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:29.834490 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:29.835168 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:29.835174 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:29.835225 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:29.835343 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:29.835663 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:29.835673 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:29.835683 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:53:29.835930 (+ 247us) client_negotiation.cc:240] Negotiation successful
0925 14:53:29.836189 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":249,"threads_started":1}
I20260925 14:53:29.838636 28061 heartbeater.cc:347] Connected to a master server at 127.17.133.254:46631
I20260925 14:53:29.838908 28061 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:29.839527 28061 heartbeater.cc:511] Master 127.17.133.254:46631 requested a full tablet report, sending...
I20260925 14:53:29.840924 27745 ts_manager.cc:194] Registered new tserver with Master: d087e0406c654b3c90c3129ad978b4d4 (127.17.133.194:44445)
W20260925 14:53:30.155589 28064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:30.155864 28064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:30.156038 28064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:30.162612 28064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:30.162698 28064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:30.162806 28064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:53:30.173450 28064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:46631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:30.174976 28064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:30.177250 28064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:30.189103 28069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:30.189904 28070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:30.190317 28072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:30.191154 28064 server_base.cc:1034] running on GCE node
I20260925 14:53:30.191797 28064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:30.192642 28064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:30.193856 28064 hybrid_clock.cc:648] HybridClock initialized: now 1790348010193823 us; error 36 us; skew 500 ppm
I20260925 14:53:30.194272 28064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:53:30.196327 28064 webserver.cc:460] Webserver started at http://127.17.133.195:46757/ using document root <none> and password file <none>
I20260925 14:53:30.197088 28064 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:30.197202 28064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:30.197530 28064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:30.200302 28064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "3b59682de2f04c97ab80c31c97579906"
format_stamp: "Formatted at 2026-09-25 14:53:30 on dist-test-slave-v5vl"
server_key: "b794ff266353a3181b6f583244adb427"
server_key_iv: "eac64b651e3cb20f4831f16284116502"
server_key_version: "encryptionkey@0"
I20260925 14:53:30.201141 28064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "3b59682de2f04c97ab80c31c97579906"
format_stamp: "Formatted at 2026-09-25 14:53:30 on dist-test-slave-v5vl"
server_key: "b794ff266353a3181b6f583244adb427"
server_key_iv: "eac64b651e3cb20f4831f16284116502"
server_key_version: "encryptionkey@0"
I20260925 14:53:30.206007 28064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:53:30.209169 28078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:30.210404 28064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:30.210572 28064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "3b59682de2f04c97ab80c31c97579906"
format_stamp: "Formatted at 2026-09-25 14:53:30 on dist-test-slave-v5vl"
server_key: "b794ff266353a3181b6f583244adb427"
server_key_iv: "eac64b651e3cb20f4831f16284116502"
server_key_version: "encryptionkey@0"
I20260925 14:53:30.210767 28064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:30.215965 28064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:30.216732 28064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:30.216979 28064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:30.217816 28064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:30.219094 28064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:30.219201 28064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:30.219331 28064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:30.219425 28064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:30.249953 28064 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:39399
I20260925 14:53:30.250000 28190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:39399 every 8 connection(s)
I20260925 14:53:30.251212 28064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:30.258127 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 28064
I20260925 14:53:30.258297 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:30.258783 17943 external_mini_cluster.cc:1428] Setting key 9dbed50c49798932314572186e879e0d
I20260925 14:53:30.259049 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.253693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:43743 (local address 127.17.133.254:46631)
0925 14:53:30.253904 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.253911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.255268 (+ 1357us) server_negotiation.cc:407] Connection header received
0925 14:53:30.256858 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.256861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.256910 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.257000 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:30.258337 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:30.258342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:30.258345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:30.258360 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:30.258410 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:30.258424 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:30.258566 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:30.258746 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 14:53:30.258810 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:53:30.260119 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.254291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:46631 (local address 127.17.133.195:43743)
0925 14:53:30.255107 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:30.255141 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:30.256625 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:30.257210 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:30.257242 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:30.257323 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:30.258034 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:30.258036 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:30.258072 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:30.258154 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:30.258526 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:30.258536 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:30.258545 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 14:53:30.258822 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 14:53:30.259053 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":254,"threads_started":1}
I20260925 14:53:30.261422 28191 heartbeater.cc:347] Connected to a master server at 127.17.133.254:46631
I20260925 14:53:30.261720 28191 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:30.262281 28191 heartbeater.cc:511] Master 127.17.133.254:46631 requested a full tablet report, sending...
I20260925 14:53:30.263535 27744 ts_manager.cc:194] Registered new tserver with Master: 3b59682de2f04c97ab80c31c97579906 (127.17.133.195:39399)
I20260925 14:53:30.274031 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:53:30.279537 27783 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:30.278146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.17.133.254:46631)
0925 14:53:30.278324 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.278331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.278817 (+ 486us) server_negotiation.cc:407] Connection header received
0925 14:53:30.278913 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.278917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.278959 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.279031 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:30.279308 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58672: BlockingRecv error: recv got EOF from 127.0.0.1:58672 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:53:30.280405 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 27804
I20260925 14:53:30.283434 28200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.281269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.17.133.193:44411)
0925 14:53:30.281690 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.281702 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.281723 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:53:30.281773 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.281778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.281935 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.282044 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:30.282825 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:30.282830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:30.282839 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:30.282850 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:30.282889 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:30.282902 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:30.282993 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:30.283062 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:53:30.283132 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":205,"threads_started":1}
I20260925 14:53:30.284504 27926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:30.364842 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 27804
I20260925 14:53:30.381458 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 27934
I20260925 14:53:30.384979 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.382354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.17.133.194:44445)
0925 14:53:30.382915 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.382922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.382948 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:53:30.383008 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.383015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.383237 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.383357 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:30.384230 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:30.384237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:30.384248 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:30.384269 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:30.384310 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:30.384324 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:30.384457 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:30.384624 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:53:30.384724 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":316,"threads_started":1}
I20260925 14:53:30.386113 28056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:30.463892 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 27934
I20260925 14:53:30.478911 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 28064
I20260925 14:53:30.482741 28204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.479826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.17.133.195:39399)
0925 14:53:30.480396 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.480408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.480435 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:53:30.480491 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.480498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.480777 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.480941 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:30.482027 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:30.482033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:30.482045 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:30.482060 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:30.482096 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:30.482109 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:30.482297 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:30.482366 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:53:30.482460 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":300,"threads_started":1}
I20260925 14:53:30.483913 28186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:30.579200 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 28064
I20260925 14:53:30.595167 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 27714
I20260925 14:53:30.596472 27775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:30.695564 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 27714
2026-09-25T14:53:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2235 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:30 dist-test-slave-v5vl krb5kdc[28209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:30 dist-test-slave-v5vl krb5kdc[28209](info): set up 2 sockets
Sep 25 14:53:30 dist-test-slave-v5vl krb5kdc[28209](info): commencing operation
krb5kdc: starting...
W20260925 14:53:31.789623 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:53:31 dist-test-slave-v5vl krb5kdc[28209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:53:32.003414 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45003
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:45003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:32.318054 28222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:32.318349 28222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:32.318439 28222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:32.325525 28222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:32.325642 28222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:32.325688 28222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:32.325744 28222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:32.325788 28222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:32.335258 28222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:45003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:32.336512 28222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:32.337893 28222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:32.353053 28228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:32.353478 28230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:32.353821 28227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:32.356730 28222 server_base.cc:1034] running on GCE node
I20260925 14:53:32.357745 28222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:32.359655 28222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:32.360900 28222 hybrid_clock.cc:648] HybridClock initialized: now 1790348012360838 us; error 63 us; skew 500 ppm
I20260925 14:53:32.361340 28222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:32 dist-test-slave-v5vl krb5kdc[28209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348012, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:32.365083 28222 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:53:32.366430 28222 webserver.cc:460] Webserver started at http://127.17.133.254:33563/ using document root <none> and password file <none>
I20260925 14:53:32.367079 28222 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:32.367195 28222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:32.367497 28222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:32.370407 28222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "eb3695a32c774a5785ccb00eb6bfc51a"
format_stamp: "Formatted at 2026-09-25 14:53:32 on dist-test-slave-v5vl"
server_key: "c54c101017a9e83eeb4c6c5c6789a3c2"
server_key_iv: "f1eb3c75a47a9d3e0a25f0f119e762c4"
server_key_version: "encryptionkey@0"
I20260925 14:53:32.371125 28222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "eb3695a32c774a5785ccb00eb6bfc51a"
format_stamp: "Formatted at 2026-09-25 14:53:32 on dist-test-slave-v5vl"
server_key: "c54c101017a9e83eeb4c6c5c6789a3c2"
server_key_iv: "f1eb3c75a47a9d3e0a25f0f119e762c4"
server_key_version: "encryptionkey@0"
I20260925 14:53:32.376096 28222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:53:32.379606 28237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:32.381069 28222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:32.381239 28222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "eb3695a32c774a5785ccb00eb6bfc51a"
format_stamp: "Formatted at 2026-09-25 14:53:32 on dist-test-slave-v5vl"
server_key: "c54c101017a9e83eeb4c6c5c6789a3c2"
server_key_iv: "f1eb3c75a47a9d3e0a25f0f119e762c4"
server_key_version: "encryptionkey@0"
I20260925 14:53:32.381421 28222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:32.394810 28222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:32.398675 28222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:32.398936 28222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:32.416342 28222 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:45003
I20260925 14:53:32.416415 28288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:45003 every 8 connection(s)
I20260925 14:53:32.417603 28222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:32.421048 28289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:32.426154 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 28222
I20260925 14:53:32.426345 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:53:32.426944 17943 external_mini_cluster.cc:1428] Setting key ef663a3a3d83c214c16646764da389e8
I20260925 14:53:32.432109 28289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a: Bootstrap starting.
I20260925 14:53:32.435738 28289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:32.436940 28289 log.cc:824] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:32.439828 28289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a: No bootstrap required, opened a new log
Sep 25 14:53:32 dist-test-slave-v5vl krb5kdc[28209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:53:32.449971 28289 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3695a32c774a5785ccb00eb6bfc51a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45003 } }
I20260925 14:53:32.450397 28289 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:32.450510 28289 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3695a32c774a5785ccb00eb6bfc51a, State: Initialized, Role: FOLLOWER
I20260925 14:53:32.450774 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:32.428877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.17.133.254:45003)
0925 14:53:32.429667 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:32.429681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:32.429729 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:53:32.431984 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:32.432009 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:32.432480 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:32.432950 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:32.434198 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:32.436198 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:32.437700 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:32.438532 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:32.438701 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:32.442455 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:32.442470 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:32.442489 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:32.442539 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:32.446421 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:32.447074 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:32.447082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:32.447091 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:32.447291 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:32.447632 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:32.447638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:32.447641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:32.448398 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:32.448731 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:32.449044 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 14:53:32.449450 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":326,"threads_started":1}
I20260925 14:53:32.451169 28289 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3695a32c774a5785ccb00eb6bfc51a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45003 } }
I20260925 14:53:32.451406 28289 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:32.451535 28289 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:32.451717 28289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:32.454640 28289 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3695a32c774a5785ccb00eb6bfc51a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45003 } }
I20260925 14:53:32.455127 28289 leader_election.cc:302] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb3695a32c774a5785ccb00eb6bfc51a; no voters:
I20260925 14:53:32.455643 28289 leader_election.cc:288] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:32.455770 28294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:32.456455 28294 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [term 1 LEADER]: Becoming Leader. State: Replica: eb3695a32c774a5785ccb00eb6bfc51a, State: Running, Role: LEADER
I20260925 14:53:32.457000 28294 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3695a32c774a5785ccb00eb6bfc51a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45003 } }
I20260925 14:53:32.457693 28289 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:32.459590 28295 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb3695a32c774a5785ccb00eb6bfc51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3695a32c774a5785ccb00eb6bfc51a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45003 } } }
I20260925 14:53:32.459914 28295 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:32.460809 28296 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb3695a32c774a5785ccb00eb6bfc51a. Latest consensus state: current_term: 1 leader_uuid: "eb3695a32c774a5785ccb00eb6bfc51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3695a32c774a5785ccb00eb6bfc51a" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 45003 } } }
I20260925 14:53:32.467875 28296 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:32.469065 28299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:32.473649 28299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:32.482316 28299 catalog_manager.cc:1347] Generated new cluster ID: 8493a348c6a44041bb38b175284b5d67
I20260925 14:53:32.482430 28299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:32.505120 28299 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:32.506426 28299 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:32.515554 28299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a: Generated new TSK 0
I20260925 14:53:32.516655 28299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:53:32.571612 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45003
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:32.887540 28315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:32.887807 28315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:32.888005 28315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:32.895061 28315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:32.895184 28315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:32.895351 28315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:53:32.904502 28315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:32.905711 28315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:32.906946 28315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:32.919386 28320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:32.920149 28321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:32.921262 28323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:32.921950 28315 server_base.cc:1034] running on GCE node
I20260925 14:53:32.922551 28315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:32.923422 28315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:32.924703 28315 hybrid_clock.cc:648] HybridClock initialized: now 1790348012924650 us; error 59 us; skew 500 ppm
I20260925 14:53:32.925001 28315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:32 dist-test-slave-v5vl krb5kdc[28209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348012, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:32.927970 28315 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:53:32.929459 28315 webserver.cc:460] Webserver started at http://127.17.133.193:45643/ using document root <none> and password file <none>
I20260925 14:53:32.930150 28315 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:32.930292 28315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:32.930588 28315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:32.933473 28315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "8f9bedfd730e41fa9d6879b7a408ac4c"
format_stamp: "Formatted at 2026-09-25 14:53:32 on dist-test-slave-v5vl"
server_key: "682cbd247950371c545d0cc5278dc94a"
server_key_iv: "6028c7fb351d05b59eee2ec60f2ccf49"
server_key_version: "encryptionkey@0"
I20260925 14:53:32.934263 28315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "8f9bedfd730e41fa9d6879b7a408ac4c"
format_stamp: "Formatted at 2026-09-25 14:53:32 on dist-test-slave-v5vl"
server_key: "682cbd247950371c545d0cc5278dc94a"
server_key_iv: "6028c7fb351d05b59eee2ec60f2ccf49"
server_key_version: "encryptionkey@0"
I20260925 14:53:32.939206 28315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:53:32.942443 28330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:32.943699 28315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:32.943869 28315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "8f9bedfd730e41fa9d6879b7a408ac4c"
format_stamp: "Formatted at 2026-09-25 14:53:32 on dist-test-slave-v5vl"
server_key: "682cbd247950371c545d0cc5278dc94a"
server_key_iv: "6028c7fb351d05b59eee2ec60f2ccf49"
server_key_version: "encryptionkey@0"
I20260925 14:53:32.944051 28315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:32.956130 28315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:32.959543 28315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:32.959803 28315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:32.960687 28315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:32.961917 28315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:32.961984 28315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:32.962059 28315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:32.962102 28315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:32.993562 28315 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:39983
I20260925 14:53:32.993603 28442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:39983 every 8 connection(s)
I20260925 14:53:32.994782 28315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:33.001008 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 28315
I20260925 14:53:33.001204 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:33.001845 17943 external_mini_cluster.cc:1428] Setting key 4206970e537a1d367e7726ef0da7e360
Sep 25 14:53:33 dist-test-slave-v5vl krb5kdc[28209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348012, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:53:33.013871 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:32.997193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:52015 (local address 127.17.133.254:45003)
0925 14:53:32.997643 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:32.997650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:32.998579 (+ 929us) server_negotiation.cc:407] Connection header received
0925 14:53:33.000264 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:33.000269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:33.000332 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:33.000438 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:33.002068 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.002950 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.004549 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.004999 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.005134 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:33.008416 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:33.008421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:33.008424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:33.008456 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:33.011065 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:33.011741 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:33.011748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:33.011752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:33.011866 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:33.012256 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:33.012263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:33.012269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:33.012516 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:33.012682 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:33.013321 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 14:53:33.013528 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":217,"threads_started":1}
I20260925 14:53:33.014910 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:32.997761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45003 (local address 127.17.133.193:52015)
0925 14:53:32.998404 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:32.998454 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:33.000035 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:33.000701 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:33.000709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:33.001229 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:33.001872 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:33.001913 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.003087 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.003092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:33.004399 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:33.004414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.005215 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.005224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:33.005467 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:33.006124 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:33.006146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:33.008202 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:33.011200 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:33.011209 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:33.011224 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:33.011626 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:33.011969 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:33.011976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:33.011981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:33.012151 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:33.012664 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:33.012671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:33.012977 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 14:53:33.013232 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 14:53:33.013734 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":249,"threads_started":1}
I20260925 14:53:33.016477 28443 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45003
I20260925 14:53:33.016795 28443 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:33.017494 28443 heartbeater.cc:511] Master 127.17.133.254:45003 requested a full tablet report, sending...
I20260925 14:53:33.019551 28254 ts_manager.cc:194] Registered new tserver with Master: 8f9bedfd730e41fa9d6879b7a408ac4c (127.17.133.193:39983)
I20260925 14:53:33.020972 28254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:52015
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:53:33.047664 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45003
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:33.359597 28449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:33.359865 28449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:33.359963 28449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:33.367202 28449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:33.367316 28449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:33.367425 28449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:53:33.377575 28449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:33.378810 28449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:33.380084 28449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:33.392683 28454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:33.392661 28455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:33.392656 28457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:33.393356 28449 server_base.cc:1034] running on GCE node
I20260925 14:53:33.393935 28449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:33.394770 28449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:33.395956 28449 hybrid_clock.cc:648] HybridClock initialized: now 1790348013395918 us; error 34 us; skew 500 ppm
I20260925 14:53:33.396198 28449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:33 dist-test-slave-v5vl krb5kdc[28209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:33.399330 28449 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:53:33.400823 28449 webserver.cc:460] Webserver started at http://127.17.133.194:41589/ using document root <none> and password file <none>
I20260925 14:53:33.401450 28449 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:33.401587 28449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:33.401875 28449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:33.404654 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "c54109b4770c4fdab6fb5869a41ef07d"
format_stamp: "Formatted at 2026-09-25 14:53:33 on dist-test-slave-v5vl"
server_key: "68c4c4617d86e813dbcc6adb74481ea7"
server_key_iv: "8539c77440f27e0345b3d4f3dda984e7"
server_key_version: "encryptionkey@0"
I20260925 14:53:33.405417 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "c54109b4770c4fdab6fb5869a41ef07d"
format_stamp: "Formatted at 2026-09-25 14:53:33 on dist-test-slave-v5vl"
server_key: "68c4c4617d86e813dbcc6adb74481ea7"
server_key_iv: "8539c77440f27e0345b3d4f3dda984e7"
server_key_version: "encryptionkey@0"
I20260925 14:53:33.410102 28449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 14:53:33.413260 28464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:33.414510 28449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:33.414674 28449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "c54109b4770c4fdab6fb5869a41ef07d"
format_stamp: "Formatted at 2026-09-25 14:53:33 on dist-test-slave-v5vl"
server_key: "68c4c4617d86e813dbcc6adb74481ea7"
server_key_iv: "8539c77440f27e0345b3d4f3dda984e7"
server_key_version: "encryptionkey@0"
I20260925 14:53:33.414846 28449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:33.426903 28449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:33.430689 28449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:33.430962 28449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:33.431774 28449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:33.433102 28449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:33.433195 28449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:33.433329 28449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:33.433399 28449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:33.465178 28449 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:39685
I20260925 14:53:33.465222 28576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:39685 every 8 connection(s)
I20260925 14:53:33.466375 28449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:33.468425 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 28449
I20260925 14:53:33.468662 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:33.469229 17943 external_mini_cluster.cc:1428] Setting key 42eeee4b57acc239f1e640f15e62348d
Sep 25 14:53:33 dist-test-slave-v5vl krb5kdc[28209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:53:33.486873 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:33.468958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:53793 (local address 127.17.133.254:45003)
0925 14:53:33.469181 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:33.469188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:33.470575 (+ 1387us) server_negotiation.cc:407] Connection header received
0925 14:53:33.472422 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:33.472427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:33.472510 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:33.472653 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:33.474108 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.474830 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.476326 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.476790 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.476919 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:33.481166 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:33.481171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:33.481174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:33.481204 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:33.483711 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:33.484560 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:33.484563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:33.484566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:33.484737 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:33.485192 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:33.485195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:33.485196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:33.485479 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:33.485593 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:33.486442 (+ 849us) server_negotiation.cc:299] Negotiation successful
0925 14:53:33.486621 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 14:53:33.488708 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:33.469566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45003 (local address 127.17.133.194:53793)
0925 14:53:33.470415 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:33.470451 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:33.472169 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:33.472882 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:33.472890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:33.473374 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:33.473946 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:33.473963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.474974 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.474986 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:33.476182 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:33.476197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.477001 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.477008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:33.477228 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:33.477989 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:33.478020 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:33.480952 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:33.483838 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:33.483848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:33.483867 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:33.484439 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:33.484845 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:33.484853 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:33.484860 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:33.485083 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:33.485620 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:33.485630 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:33.486076 (+ 446us) client_negotiation.cc:769] Sending connection context
0925 14:53:33.486380 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 14:53:33.487000 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":373,"threads_started":1}
I20260925 14:53:33.490284 28577 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45003
I20260925 14:53:33.490597 28577 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:33.491309 28577 heartbeater.cc:511] Master 127.17.133.254:45003 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
I20260925 14:53:33.492823 28254 ts_manager.cc:194] Registered new tserver with Master: c54109b4770c4fdab6fb5869a41ef07d (127.17.133.194:39685)
I20260925 14:53:33.493716 28254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:53793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:53:33.521572 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45003
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:33.838961 28582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:33.839221 28582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:33.839309 28582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:33.846076 28582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:33.846164 28582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:33.846275 28582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:53:33.855566 28582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:45003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:53:33.856840 28582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:33.858145 28582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:33.870971 28588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:33.871148 28587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:33.871083 28590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:33.872095 28582 server_base.cc:1034] running on GCE node
I20260925 14:53:33.872768 28582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:33.873646 28582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:33.874861 28582 hybrid_clock.cc:648] HybridClock initialized: now 1790348013874826 us; error 53 us; skew 500 ppm
I20260925 14:53:33.875155 28582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:53:33 dist-test-slave-v5vl krb5kdc[28209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:33.878105 28582 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:53:33.879422 28582 webserver.cc:460] Webserver started at http://127.17.133.195:45653/ using document root <none> and password file <none>
I20260925 14:53:33.880065 28582 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:33.880204 28582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:33.880497 28582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:33.883415 28582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "ee43119f950b4e6884be5d79843e6dd5"
format_stamp: "Formatted at 2026-09-25 14:53:33 on dist-test-slave-v5vl"
server_key: "26588a2bf0370e33041a71de4b24d240"
server_key_iv: "7e6ef8f52e4f93c098aa4cb7a5b164ae"
server_key_version: "encryptionkey@0"
I20260925 14:53:33.884145 28582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "ee43119f950b4e6884be5d79843e6dd5"
format_stamp: "Formatted at 2026-09-25 14:53:33 on dist-test-slave-v5vl"
server_key: "26588a2bf0370e33041a71de4b24d240"
server_key_iv: "7e6ef8f52e4f93c098aa4cb7a5b164ae"
server_key_version: "encryptionkey@0"
I20260925 14:53:33.888778 28582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 14:53:33.891933 28597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:33.893157 28582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:33.893312 28582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "ee43119f950b4e6884be5d79843e6dd5"
format_stamp: "Formatted at 2026-09-25 14:53:33 on dist-test-slave-v5vl"
server_key: "26588a2bf0370e33041a71de4b24d240"
server_key_iv: "7e6ef8f52e4f93c098aa4cb7a5b164ae"
server_key_version: "encryptionkey@0"
I20260925 14:53:33.893474 28582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:33.903800 28582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:33.907356 28582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:33.907646 28582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:33.908485 28582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:33.909869 28582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:33.909968 28582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:33.910109 28582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:33.910182 28582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:33.941987 28582 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:45291
I20260925 14:53:33.942073 28709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:45291 every 8 connection(s)
I20260925 14:53:33.943218 28582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:33.952405 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 28582
I20260925 14:53:33.952651 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:33.953193 17943 external_mini_cluster.cc:1428] Setting key 0c72a001da1d24192e305bf4610ef86a
Sep 25 14:53:33 dist-test-slave-v5vl krb5kdc[28209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:53:33.962455 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:33.945825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:47929 (local address 127.17.133.254:45003)
0925 14:53:33.946029 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:33.946035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:33.947204 (+ 1169us) server_negotiation.cc:407] Connection header received
0925 14:53:33.949156 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:33.949160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:33.949222 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:33.949308 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:33.950637 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.951354 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.952869 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.953477 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.953606 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:33.957633 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:33.957638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:33.957641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:33.957671 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:33.959993 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:33.960677 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:33.960680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:33.960683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:33.960797 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:33.961142 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:33.961144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:33.961146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:33.961392 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:33.961495 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:33.962063 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 14:53:33.962209 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:53:33.963400 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:33.946371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:45003 (local address 127.17.133.195:47929)
0925 14:53:33.947027 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:33.947071 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:33.948922 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:33.949494 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:33.949503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:33.949959 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:33.950491 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:33.950507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.951471 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.951475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:33.952699 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:33.952715 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.953680 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.953689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:33.953962 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:33.954729 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:33.954749 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:33.957449 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:33.960109 (+ 2660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:33.960115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:33.960128 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:33.960514 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:33.960883 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:33.960886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:33.960888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:33.961048 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:33.961485 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:33.961490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:33.961770 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 14:53:33.961977 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 14:53:33.962336 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":256,"threads_started":1}
I20260925 14:53:33.964813 28710 heartbeater.cc:347] Connected to a master server at 127.17.133.254:45003
I20260925 14:53:33.965073 28710 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:33.965693 28710 heartbeater.cc:511] Master 127.17.133.254:45003 requested a full tablet report, sending...
I20260925 14:53:33.967162 28254 ts_manager.cc:194] Registered new tserver with Master: ee43119f950b4e6884be5d79843e6dd5 (127.17.133.195:45291)
I20260925 14:53:33.968248 28254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:47929
I20260925 14:53:33.968624 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:33.984369 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:33.972963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.17.133.254:45003)
0925 14:53:33.973171 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:33.973179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:33.973565 (+ 386us) server_negotiation.cc:407] Connection header received
0925 14:53:33.973680 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:33.973684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:33.973736 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:33.973818 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:33.975001 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.975694 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.976930 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:33.977327 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:33.977420 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:33.979427 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:33.979431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:33.979434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:33.979462 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:33.982042 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:33.982793 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:33.982797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:33.982799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:33.982915 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:33.983225 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:33.983228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:33.983230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:33.983538 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:33.983691 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:33.983943 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 14:53:33.984082 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:53:33.988292 28254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:33.995323 28254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:34.023833 28443 heartbeater.cc:503] Master 127.17.133.254:45003 was elected leader, sending a full tablet report...
I20260925 14:53:34.030118 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.008865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.17.133.195:45291)
0925 14:53:34.020244 (+ 11379us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:34.020257 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:34.021786 (+ 1529us) server_negotiation.cc:407] Connection header received
0925 14:53:34.021902 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:34.021913 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:34.022121 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:34.022305 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:34.023089 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.024575 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.027544 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.028750 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.028928 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.029119 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:34.029604 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 14:53:34.029776 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11147,"thread_start_us":10957,"threads_started":1}
I20260925 14:53:34.030119 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.007069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:39983 (local address 127.0.0.1:36914)
0925 14:53:34.016499 (+ 9430us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.016558 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:34.016863 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:34.021952 (+ 5089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:34.021961 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:34.021986 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:34.022508 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.022523 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.025470 (+ 2947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.025475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.027151 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.027163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.029070 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.029077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.029262 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.029318 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 14:53:34.029443 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 14:53:34.029641 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9062,"mutex_wait_us":58,"thread_start_us":12302,"threads_started":1}
I20260925 14:53:34.030436 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.007272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.17.133.193:39983)
0925 14:53:34.021206 (+ 13934us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:34.021217 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:34.021252 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:53:34.021356 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:34.021364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:34.021630 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:34.022507 (+ 877us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:34.022756 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.025311 (+ 2555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.027307 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.028828 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.029744 (+ 916us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.029896 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:34.029959 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:53:34.030116 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13741,"thread_start_us":13606,"threads_started":1}
I20260925 14:53:34.030691 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.007439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:39685 (local address 127.0.0.1:42738)
0925 14:53:34.019642 (+ 12203us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.019661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:34.019785 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:34.021863 (+ 2078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:34.021870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:34.021898 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:34.022415 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.022431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.024439 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.024444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.027341 (+ 2897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.027355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.029419 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.029428 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.029643 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.029670 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 14:53:34.029747 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:53:34.029883 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12072,"mutex_wait_us":523,"thread_start_us":13768,"threads_started":1}
I20260925 14:53:34.031126 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.008296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.17.133.194:39685)
0925 14:53:34.020827 (+ 12531us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:34.020839 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:34.020879 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:53:34.021020 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:34.021033 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:34.021361 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:34.021574 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:34.022638 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.024273 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.027500 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.028959 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.029112 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.029268 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:34.030617 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0925 14:53:34.030806 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12315,"thread_start_us":12122,"threads_started":1}
I20260925 14:53:34.031404 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.007653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:45291 (local address 127.0.0.1:47528)
0925 14:53:34.021487 (+ 13834us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.021506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:34.021650 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:34.022580 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:34.022585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:34.022611 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:34.022922 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.022929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.024787 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.024791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.027341 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.027355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.029266 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.029271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.029412 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.029435 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:53:34.029552 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 14:53:34.031052 (+ 1500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13698,"mutex_wait_us":1390,"thread_start_us":8291,"threads_started":1}
I20260925 14:53:34.034564 28512 tablet_service.cc:1467] Processing CreateTablet for tablet 09f77de21c304a5fa943b1b4b82ae375 (DEFAULT_TABLE table=test-table [id=561246b0717c4b6bbc04dddea96e7971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:34.034382 28378 tablet_service.cc:1467] Processing CreateTablet for tablet 09f77de21c304a5fa943b1b4b82ae375 (DEFAULT_TABLE table=test-table [id=561246b0717c4b6bbc04dddea96e7971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:34.034550 28645 tablet_service.cc:1467] Processing CreateTablet for tablet 09f77de21c304a5fa943b1b4b82ae375 (DEFAULT_TABLE table=test-table [id=561246b0717c4b6bbc04dddea96e7971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:34.035712 28645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f77de21c304a5fa943b1b4b82ae375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:34.035755 28512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f77de21c304a5fa943b1b4b82ae375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:34.035739 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f77de21c304a5fa943b1b4b82ae375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:34.046516 28728 tablet_bootstrap.cc:492] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: Bootstrap starting.
I20260925 14:53:34.049547 28729 tablet_bootstrap.cc:492] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: Bootstrap starting.
I20260925 14:53:34.050459 28728 tablet_bootstrap.cc:654] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:34.051926 28728 log.cc:824] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:34.052011 28730 tablet_bootstrap.cc:492] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: Bootstrap starting.
I20260925 14:53:34.053225 28729 tablet_bootstrap.cc:654] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:34.054401 28729 log.cc:824] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:34.054970 28730 tablet_bootstrap.cc:654] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:34.055243 28728 tablet_bootstrap.cc:492] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: No bootstrap required, opened a new log
I20260925 14:53:34.055498 28728 ts_tablet_manager.cc:1397] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260925 14:53:34.056051 28730 log.cc:824] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:34.057404 28729 tablet_bootstrap.cc:492] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: No bootstrap required, opened a new log
I20260925 14:53:34.057660 28729 ts_tablet_manager.cc:1397] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260925 14:53:34.058511 28730 tablet_bootstrap.cc:492] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: No bootstrap required, opened a new log
I20260925 14:53:34.058801 28730 ts_tablet_manager.cc:1397] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260925 14:53:34.066212 28730 raft_consensus.cc:348] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } }
I20260925 14:53:34.066553 28730 raft_consensus.cc:374] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:34.066363 28728 raft_consensus.cc:348] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } }
I20260925 14:53:34.066637 28730 raft_consensus.cc:729] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9bedfd730e41fa9d6879b7a408ac4c, State: Initialized, Role: FOLLOWER
I20260925 14:53:34.066748 28728 raft_consensus.cc:374] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:34.066864 28728 raft_consensus.cc:729] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee43119f950b4e6884be5d79843e6dd5, State: Initialized, Role: FOLLOWER
I20260925 14:53:34.067319 28730 consensus_queue.cc:260] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } }
I20260925 14:53:34.067466 28728 consensus_queue.cc:260] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } }
I20260925 14:53:34.067761 28729 raft_consensus.cc:348] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } }
I20260925 14:53:34.068216 28729 raft_consensus.cc:374] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:34.069332 28710 heartbeater.cc:503] Master 127.17.133.254:45003 was elected leader, sending a full tablet report...
I20260925 14:53:34.069893 28728 ts_tablet_manager.cc:1428] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260925 14:53:34.068328 28729 raft_consensus.cc:729] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54109b4770c4fdab6fb5869a41ef07d, State: Initialized, Role: FOLLOWER
I20260925 14:53:34.070652 28729 consensus_queue.cc:260] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } }
I20260925 14:53:34.069142 28730 ts_tablet_manager.cc:1428] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260925 14:53:34.072237 28577 heartbeater.cc:503] Master 127.17.133.254:45003 was elected leader, sending a full tablet report...
I20260925 14:53:34.072866 28729 ts_tablet_manager.cc:1428] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260925 14:53:34.075428 28734 raft_consensus.cc:482] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:34.075697 28734 raft_consensus.cc:504] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } }
I20260925 14:53:34.077349 28734 leader_election.cc:288] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c54109b4770c4fdab6fb5869a41ef07d (127.17.133.194:39685), ee43119f950b4e6884be5d79843e6dd5 (127.17.133.195:45291)
I20260925 14:53:34.083673 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.077622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:60421 (local address 127.17.133.194:39685)
0925 14:53:34.077827 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:34.077835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:34.078299 (+ 464us) server_negotiation.cc:407] Connection header received
0925 14:53:34.078585 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:34.078589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:34.078642 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:34.078730 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:34.079257 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.080014 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.081935 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.082729 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.082926 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.083030 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:34.083285 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 14:53:34.083407 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:53:34.084084 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.078289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:58335 (local address 127.17.133.195:45291)
0925 14:53:34.078574 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:34.078582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:34.078599 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:53:34.078794 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:34.078799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:34.078866 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:34.078962 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:34.079532 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.080239 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.082078 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.082965 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.083129 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.083206 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:34.083731 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 14:53:34.083849 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260925 14:53:34.084097 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.077514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:39685 (local address 127.17.133.193:60421)
0925 14:53:34.078190 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.078210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:34.078421 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:34.078757 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:34.078761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:34.078787 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:34.079125 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.079135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.080133 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.080141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.081747 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.081784 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.082965 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.082971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.083133 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.083156 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:53:34.083262 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:53:34.083408 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":5,"thread_start_us":347,"threads_started":1}
I20260925 14:53:34.084815 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.077772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:45291 (local address 127.17.133.193:58335)
0925 14:53:34.078516 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.078534 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:34.078661 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:34.078975 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:34.078979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:34.079007 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:34.079392 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.079399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.080372 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.080381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.081929 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:34.081946 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.083371 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.083379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:34.083552 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.083580 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 14:53:34.084350 (+ 770us) client_negotiation.cc:240] Negotiation successful
0925 14:53:34.084481 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":18,"thread_start_us":437,"threads_started":1}
I20260925 14:53:34.085026 28532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77de21c304a5fa943b1b4b82ae375" candidate_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54109b4770c4fdab6fb5869a41ef07d" is_pre_election: true
I20260925 14:53:34.085533 28532 raft_consensus.cc:2454] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9bedfd730e41fa9d6879b7a408ac4c in term 0.
I20260925 14:53:34.085738 28665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77de21c304a5fa943b1b4b82ae375" candidate_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee43119f950b4e6884be5d79843e6dd5" is_pre_election: true
I20260925 14:53:34.086185 28665 raft_consensus.cc:2454] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9bedfd730e41fa9d6879b7a408ac4c in term 0.
I20260925 14:53:34.086190 28334 leader_election.cc:302] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9bedfd730e41fa9d6879b7a408ac4c, c54109b4770c4fdab6fb5869a41ef07d; no voters:
I20260925 14:53:34.086547 28734 raft_consensus.cc:2781] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:34.086699 28734 raft_consensus.cc:482] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:34.086824 28734 raft_consensus.cc:3037] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:34.088980 28734 raft_consensus.cc:504] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } }
I20260925 14:53:34.089588 28734 leader_election.cc:288] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [CANDIDATE]: Term 1 election: Requested vote from peers c54109b4770c4fdab6fb5869a41ef07d (127.17.133.194:39685), ee43119f950b4e6884be5d79843e6dd5 (127.17.133.195:45291)
I20260925 14:53:34.089907 28532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77de21c304a5fa943b1b4b82ae375" candidate_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54109b4770c4fdab6fb5869a41ef07d"
I20260925 14:53:34.090039 28665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77de21c304a5fa943b1b4b82ae375" candidate_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee43119f950b4e6884be5d79843e6dd5"
I20260925 14:53:34.090093 28532 raft_consensus.cc:3037] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:34.090225 28665 raft_consensus.cc:3037] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:34.093218 28532 raft_consensus.cc:2454] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9bedfd730e41fa9d6879b7a408ac4c in term 1.
I20260925 14:53:34.093218 28665 raft_consensus.cc:2454] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9bedfd730e41fa9d6879b7a408ac4c in term 1.
I20260925 14:53:34.093575 28332 leader_election.cc:302] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9bedfd730e41fa9d6879b7a408ac4c, ee43119f950b4e6884be5d79843e6dd5; no voters:
I20260925 14:53:34.093854 28734 raft_consensus.cc:2781] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:34.094276 28734 raft_consensus.cc:686] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 1 LEADER]: Becoming Leader. State: Replica: 8f9bedfd730e41fa9d6879b7a408ac4c, State: Running, Role: LEADER
I20260925 14:53:34.094699 28734 consensus_queue.cc:237] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } }
I20260925 14:53:34.098987 28252 catalog_manager.cc:5495] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9bedfd730e41fa9d6879b7a408ac4c (127.17.133.193). New cstate: current_term: 1 leader_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9bedfd730e41fa9d6879b7a408ac4c" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 39983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:34.130347 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.125142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.17.133.193:39983)
0925 14:53:34.125350 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:34.125356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:34.125467 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:53:34.125707 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:34.125711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:34.125781 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:34.125927 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:34.126289 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.126961 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.128257 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.128725 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.128855 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.129335 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:34.129795 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:34.129947 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:34.130024 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:53:34.130096 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:53:34.137953 28665 raft_consensus.cc:1261] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [term 1 FOLLOWER]: Refusing update from remote peer 8f9bedfd730e41fa9d6879b7a408ac4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:34.137953 28532 raft_consensus.cc:1261] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [term 1 FOLLOWER]: Refusing update from remote peer 8f9bedfd730e41fa9d6879b7a408ac4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:34.138952 28739 consensus_queue.cc:1035] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee43119f950b4e6884be5d79843e6dd5" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:34.139281 28734 consensus_queue.cc:1035] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54109b4770c4fdab6fb5869a41ef07d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:34.149817 28742 mvcc.cc:205] Tried to move back new op lower bound from 7333265465898749952 to 7333265465735127040. Current Snapshot: MvccSnapshot[applied={T|T < 7333265465898749952}]
I20260925 14:53:34.162473 28253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:34.162742 28253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:34.165926 28253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb3695a32c774a5785ccb00eb6bfc51a: Sending DeleteTablet for 3 replicas of tablet 09f77de21c304a5fa943b1b4b82ae375
I20260925 14:53:34.166822 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f77de21c304a5fa943b1b4b82ae375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:34 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:36914
I20260925 14:53:34.166996 28512 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f77de21c304a5fa943b1b4b82ae375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:34 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:42738
I20260925 14:53:34.167651 28645 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f77de21c304a5fa943b1b4b82ae375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:34 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:47528
I20260925 14:53:34.168478 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 28315
I20260925 14:53:34.182561 28751 tablet_replica.cc:330] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: stopping tablet replica
I20260925 14:53:34.183355 28752 tablet_replica.cc:330] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: stopping tablet replica
I20260925 14:53:34.183753 28752 raft_consensus.cc:2229] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:34.184197 28752 raft_consensus.cc:2258] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:34.182827 28753 tablet_replica.cc:330] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: stopping tablet replica
I20260925 14:53:34.184844 28751 raft_consensus.cc:2229] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:34.184935 28753 raft_consensus.cc:2229] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:34.185458 28753 raft_consensus.cc:2258] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:34.185465 28751 raft_consensus.cc:2258] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:34.187572 28753 ts_tablet_manager.cc:1905] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:34.187649 28752 ts_tablet_manager.cc:1905] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 14:53:34 dist-test-slave-v5vl krb5kdc[28209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:53:34.194584 28753 ts_tablet_manager.cc:1918] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:34.194676 28752 ts_tablet_manager.cc:1918] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:34.194839 28752 log.cc:1196] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/09f77de21c304a5fa943b1b4b82ae375
I20260925 14:53:34.194855 28753 log.cc:1196] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/09f77de21c304a5fa943b1b4b82ae375
I20260925 14:53:34.195374 28752 ts_tablet_manager.cc:1939] T 09f77de21c304a5fa943b1b4b82ae375 P c54109b4770c4fdab6fb5869a41ef07d: Deleting consensus metadata
I20260925 14:53:34.195544 28753 ts_tablet_manager.cc:1939] T 09f77de21c304a5fa943b1b4b82ae375 P ee43119f950b4e6884be5d79843e6dd5: Deleting consensus metadata
I20260925 14:53:34.196421 28751 ts_tablet_manager.cc:1905] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:34.197355 28239 catalog_manager.cc:4841] TS ee43119f950b4e6884be5d79843e6dd5 (127.17.133.195:45291): tablet 09f77de21c304a5fa943b1b4b82ae375 (table test-table [id=561246b0717c4b6bbc04dddea96e7971]) successfully deleted
I20260925 14:53:34.197710 28241 catalog_manager.cc:4841] TS c54109b4770c4fdab6fb5869a41ef07d (127.17.133.194:39685): tablet 09f77de21c304a5fa943b1b4b82ae375 (table test-table [id=561246b0717c4b6bbc04dddea96e7971]) successfully deleted
I20260925 14:53:34.201997 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.172412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.17.133.193:39983)
0925 14:53:34.172696 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:34.172702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:34.182632 (+ 9930us) server_negotiation.cc:407] Connection header received
0925 14:53:34.183246 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:34.183252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:34.183341 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:34.183461 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:34.185239 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.186458 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.188447 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.189121 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.189216 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.194365 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:34.194379 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:34.194399 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:34.194454 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:34.198669 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:34.199511 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:34.199520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:34.199524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:34.199695 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:34.200169 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:34.200174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:34.200178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:34.200693 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:34.201012 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:34.201332 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 14:53:34.201556 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 14:53:34.202557 28751 ts_tablet_manager.cc:1918] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:34.202605 28438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:34.206614 28751 log.cc:1196] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/09f77de21c304a5fa943b1b4b82ae375
I20260925 14:53:34.299808 28751 ts_tablet_manager.cc:1939] T 09f77de21c304a5fa943b1b4b82ae375 P 8f9bedfd730e41fa9d6879b7a408ac4c: Deleting consensus metadata
I20260925 14:53:34.299855 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 28315
I20260925 14:53:34.317131 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 28449
W20260925 14:53:34.317099 28240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.133.193:39983 (error 108)
I20260925 14:53:34.318944 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.318516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:39983 (local address 127.0.0.1:36932)
0925 14:53:34.318669 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.318762 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 14:53:34.319181 28240 catalog_manager.cc:4584] TS 8f9bedfd730e41fa9d6879b7a408ac4c (127.17.133.193:39983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09f77de21c304a5fa943b1b4b82ae375: Network error: Client connection negotiation failed: client connection to 127.17.133.193:39983: connect: Connection refused (error 111)
Sep 25 14:53:34 dist-test-slave-v5vl krb5kdc[28209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:53:34.331494 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.317874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.17.133.194:39685)
0925 14:53:34.318056 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:34.318063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:34.318279 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:53:34.318493 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:34.318502 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:34.318564 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:34.318666 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:34.320097 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.320894 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.322058 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.322443 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.322573 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.326289 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:34.326301 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:34.326314 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:34.326342 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:34.328914 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:34.329545 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:34.329553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:34.329555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:34.329663 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:34.330002 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:34.330009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:34.330012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:34.330375 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:34.330660 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:34.330944 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 14:53:34.331143 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:53:34.332162 28572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:34.372658 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.372125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:39983 (local address 127.0.0.1:36938)
0925 14:53:34.372332 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.372445 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 14:53:34.373657 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.373287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:39983 (local address 127.0.0.1:36948)
0925 14:53:34.373441 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.373531 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 14:53:34.419214 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 28449
I20260925 14:53:34.434131 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.433597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:39983 (local address 127.0.0.1:36960)
0925 14:53:34.433826 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.433936 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 14:53:34.435190 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.434758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:39983 (local address 127.0.0.1:36970)
0925 14:53:34.434960 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.435046 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 14:53:34.436645 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 28582
Sep 25 14:53:34 dist-test-slave-v5vl krb5kdc[28209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:53:34.450182 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.437378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.17.133.195:45291)
0925 14:53:34.437539 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:34.437543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:34.437561 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:53:34.437633 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:34.437637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:34.437702 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:34.437778 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:34.438927 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.439601 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.440993 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:34.441678 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:34.441777 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:34.445477 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:34.445489 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:34.445504 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:34.445536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:34.447950 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:34.448481 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:34.448488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:34.448490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:34.448678 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:34.449036 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:34.449039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:34.449041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:34.449371 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:34.449574 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:34.449795 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:53:34.449949 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:53:34.450829 28705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:34.515815 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.515335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:39983 (local address 127.0.0.1:36978)
0925 14:53:34.515531 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.515651 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 14:53:34.516847 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:34.516440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:39983 (local address 127.0.0.1:36990)
0925 14:53:34.516691 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:34.516754 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 14:53:34.540231 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 28582
I20260925 14:53:34.556404 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 28222
I20260925 14:53:34.557449 28284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:53:34.662406 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 28222
2026-09-25T14:53:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3968 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:34 dist-test-slave-v5vl krb5kdc[28762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:34 dist-test-slave-v5vl krb5kdc[28762](info): set up 2 sockets
Sep 25 14:53:34 dist-test-slave-v5vl krb5kdc[28762](info): commencing operation
krb5kdc: starting...
W20260925 14:53:35.741609 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:53:35 dist-test-slave-v5vl krb5kdc[28762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.133.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.133.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/rangeradmin_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/rangeradmin_127.17.133.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.133.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.133.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/rangerlookup_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/rangerlookup_127.17.133.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.133.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.133.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/HTTP_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/HTTP_127.17.133.212@KRBTEST.COM.keytab.
I20260925 14:53:36.048578 17943 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNtxnNj/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/postgres -l logfile start
2026-09-25 14:53:38.324 UTC [28792] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:53:38.324 UTC [28792] LOG: listening on IPv4 address "127.17.133.212", port 44633
2026-09-25 14:53:38.327 UTC [28792] LOG: listening on Unix socket "/tmp/.s.PGSQL.44633"
2026-09-25 14:53:38.348 UTC [28795] LOG: database system was shut down at 2026-09-25 14:53:37 UTC
2026-09-25 14:53:38.351 UTC [28792] LOG: database system is ready to accept connections
I20260925 14:53:39.313496 17943 mini_postgres.cc:96] Postgres bound to 44633
2026-09-25 14:53:39.329 UTC [28803] FATAL: database "slave" does not exist
127.17.133.212:44633 - accepting connections
I20260925 14:53:39.330506 17943 mini_ranger.cc:161] Starting Ranger...
I20260925 14:53:39.353289 17943 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:53:39.770665 17943 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:53:39.770821 17943 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-admin
2026-09-25 14:53:40,047 [I] DB FLAVOR :POSTGRES
2026-09-25 14:53:40,047 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:53:40,047 [I] Checking connection..
2026-09-25 14:53:40,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:53:40,286 [I] Checking connection passed.
2026-09-25 14:53:40,286 [I] --------- Verifying version history table ---------
2026-09-25 14:53:40,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:53:40,535 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:53:40,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:53:40,803 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:53:40,803 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:53:40,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:53:41.049 UTC [28867] WARNING: there is no transaction in progress
2026-09-25 14:53:41,055 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:53:41,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:53:41,290 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:53:41,290 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:53:41,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:53:41,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:53:41,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v5vl.c.gcp-upstream.internal','N') ;"
2026-09-25 14:53:42,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:53:42,271 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:53:42,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:53:42,510 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:53:42,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:53:42,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:53:42,979 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:53:42,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:53:43.890 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:43.905 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:43.919 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:43.933 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:43.947 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:43.963 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.025 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.032 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.045 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.055 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.065 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.077 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.086 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.093 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.098 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44.104 UTC [28975] WARNING: there is no transaction in progress
2026-09-25 14:53:44,663 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:53:44,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:53:44,907 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:53:44,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:53:45,141 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:53:45,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:53:45,367 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:53:45,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:53:45,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:53:45,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v5vl.c.gcp-upstream.internal';"
2026-09-25 14:53:46,082 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:53:46,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:53:46,327 [I] DB_PATCHES have already been applied
I20260925 14:53:46.332919 17943 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNtxnNj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:53:46.333051 17943 mini_ranger.cc:193] Using host: 127.17.133.212
I20260925 14:53:46.347579 17943 mini_ranger.cc:227] Ranger admin URL: http://127.17.133.212:39747
Sep 25, 2026 2:53:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:53:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:53:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:53:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:53:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:53:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:53:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:53:47.480676 17943 mini_ranger.cc:160] Time spent starting Ranger: real 8.150s user 0.008s sys 0.029s
Sep 25, 2026 2:54:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:54:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 2:54:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:54:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:54:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:54:07.338941 17943 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.133.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.133.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/rangerkms_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/rangerkms_127.17.133.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.133.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.133.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.133.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/HTTP_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.133.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/HTTP_127.17.133.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:54:07 dist-test-slave-v5vl krb5kdc[28762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:54:07.503393 17943 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:54:07.524387 17943 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:54:08.333887 17943 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:54:08.334029 17943 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-kms
I20260925 14:54:08.334190 17943 mini_ranger_kms.cc:227] Using postgres at 127.17.133.212:44633
2026-09-25 14:54:08,756 [I] DB FLAVOR :POSTGRES
2026-09-25 14:54:08,757 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:54:08,757 [I] Checking connection
2026-09-25 14:54:08,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:54:08,996 [I] connection success
2026-09-25 14:54:08,996 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:54:08,996 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:54:08,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:54:09,219 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:54:09,220 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:54:09,220 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:54:09,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:44633/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:54:09,468 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:54:09.938203 17943 mini_ranger_kms.cc:310] Created kms service
I20260925 14:54:10.149296 17943 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:54:10.194588 17943 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:54:10 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:54:10.560426 17943 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:54:10.560761 17943 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:54:10.560823 17943 mini_ranger_kms.cc:240] Using host: 127.17.133.212
I20260925 14:54:10.581529 17943 mini_ranger_kms.cc:276] Ranger KMS PID: 29155
I20260925 14:54:10.581727 17943 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.133.212:47069
Sep 25, 2026 2:54:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:54:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:54:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:54:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 14:54:12 dist-test-slave-v5vl krb5kdc[28762](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348052, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.133.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:12.833314 17943 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.330s user 0.111s sys 0.331s
I20260925 14:54:12.833575 17943 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:54:12.833835 17943 mini_ranger_kms.cc:401] 127.17.133.212:47069/kms/v1/keys
Sep 25 14:54:13 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348052, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.133.212@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
Sep 25 14:54:13 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348052, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.133.212@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
Sep 25 14:54:15 dist-test-slave-v5vl krb5kdc[28762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:54:15.680030 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:39189
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:39189
--ranger_config_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:47069/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:16.008153 29206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:16.008500 29206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:16.008684 29206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:16.015748 29206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:16.015904 29206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:16.016024 29206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:16.016119 29206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:16.016209 29206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:16.026602 29206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.133.212:47069/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:39189
--ranger_config_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:54:16.028151 29206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:16.029443 29206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:16.040609 29212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:16.040692 29211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:16.040716 29214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:16.041427 29206 server_base.cc:1034] running on GCE node
I20260925 14:54:16.042130 29206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:16.043385 29206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:16.044616 29206 hybrid_clock.cc:648] HybridClock initialized: now 1790348056044544 us; error 33 us; skew 500 ppm
I20260925 14:54:16.044970 29206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:54:16 dist-test-slave-v5vl krb5kdc[28762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348056, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:16.051537 29206 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:54:16.053047 29206 webserver.cc:460] Webserver started at http://127.17.133.254:44501/ using document root <none> and password file <none>
I20260925 14:54:16.053690 29206 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:16.053828 29206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:16.054136 29206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:54:16 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348056, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:54:16.101593 29206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "deb9a5c44437462e847b8137e229bc41"
format_stamp: "Formatted at 2026-09-25 14:54:16 on dist-test-slave-v5vl"
server_key: "d38cb230b1c1128481e0ce79d6597ad7"
server_key_iv: "cff9a467ed91e6973aeeb9a653059b89"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:16.102412 29206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "deb9a5c44437462e847b8137e229bc41"
format_stamp: "Formatted at 2026-09-25 14:54:16 on dist-test-slave-v5vl"
server_key: "d38cb230b1c1128481e0ce79d6597ad7"
server_key_iv: "cff9a467ed91e6973aeeb9a653059b89"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:16.108078 29206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 14:54:16.123660 29221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:16.125223 29206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:16.125414 29206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "deb9a5c44437462e847b8137e229bc41"
format_stamp: "Formatted at 2026-09-25 14:54:16 on dist-test-slave-v5vl"
server_key: "d38cb230b1c1128481e0ce79d6597ad7"
server_key_iv: "cff9a467ed91e6973aeeb9a653059b89"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:16.125655 29206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:16.140476 29206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:16.144227 29206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:16.144534 29206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:16.163218 29206 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:39189
I20260925 14:54:16.163239 29272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:39189 every 8 connection(s)
I20260925 14:54:16.164731 29206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:16.168941 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:16.174073 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 29206
I20260925 14:54:16.174310 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:54:16.175146 17943 external_mini_cluster.cc:1428] Setting key f9a6981a9beb38aeabcae453fc7350fd
I20260925 14:54:16.183221 29273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41: Bootstrap starting.
I20260925 14:54:16.186601 29273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:16.187518 29273 log.cc:824] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:16.190124 29273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41: No bootstrap required, opened a new log
Sep 25 14:54:16 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:54:16.197641 29273 raft_consensus.cc:348] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb9a5c44437462e847b8137e229bc41" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39189 } }
I20260925 14:54:16.197983 29273 raft_consensus.cc:374] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:16.198108 29273 raft_consensus.cc:729] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb9a5c44437462e847b8137e229bc41, State: Initialized, Role: FOLLOWER
I20260925 14:54:16.198633 29273 consensus_queue.cc:260] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb9a5c44437462e847b8137e229bc41" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39189 } }
I20260925 14:54:16.199293 29273 raft_consensus.cc:388] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:16.199436 29273 raft_consensus.cc:482] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:16.199577 29276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:16.177343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.17.133.254:39189)
0925 14:54:16.180537 (+ 3194us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:16.180549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:16.180654 (+ 105us) server_negotiation.cc:407] Connection header received
0925 14:54:16.182342 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:16.182364 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:16.182744 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:16.183127 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:16.184378 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.186111 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.187378 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.188024 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.188229 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:16.192127 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:16.192134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:16.192147 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:16.192189 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:16.195201 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:16.195889 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:16.195895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:16.195900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:16.196055 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:16.196416 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:16.196421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:16.196426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:16.197003 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:16.197290 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:16.197711 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 14:54:16.198092 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2963,"thread_start_us":2733,"threads_started":1}
I20260925 14:54:16.199621 29273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:16.202212 29273 raft_consensus.cc:504] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb9a5c44437462e847b8137e229bc41" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39189 } }
I20260925 14:54:16.202648 29273 leader_election.cc:302] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deb9a5c44437462e847b8137e229bc41; no voters:
I20260925 14:54:16.203162 29273 leader_election.cc:288] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:16.203279 29278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:16.203923 29278 raft_consensus.cc:686] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [term 1 LEADER]: Becoming Leader. State: Replica: deb9a5c44437462e847b8137e229bc41, State: Running, Role: LEADER
I20260925 14:54:16.204428 29278 consensus_queue.cc:237] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb9a5c44437462e847b8137e229bc41" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39189 } }
I20260925 14:54:16.204999 29273 sys_catalog.cc:564] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:16.206974 29279 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deb9a5c44437462e847b8137e229bc41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb9a5c44437462e847b8137e229bc41" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39189 } } }
I20260925 14:54:16.206876 29280 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deb9a5c44437462e847b8137e229bc41. Latest consensus state: current_term: 1 leader_uuid: "deb9a5c44437462e847b8137e229bc41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb9a5c44437462e847b8137e229bc41" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39189 } } }
I20260925 14:54:16.207355 29279 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:16.207366 29280 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:16.209669 29284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:16.213394 29284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:16.221944 29284 catalog_manager.cc:1347] Generated new cluster ID: fa24dd8a92f34bed890d639912a1f8ec
I20260925 14:54:16.222040 29284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:16.226091 29273 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:54:16.247934 29284 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:16.250133 29284 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:16.259138 29284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41: Generated new TSK 0
I20260925 14:54:16.260192 29284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:54:16.979880 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:47069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:39189
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:54:17 dist-test-slave-v5vl krb5kdc[28762](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348057, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 14:54:17.431105 29318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:17.431516 29318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:17.431680 29318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:17.443790 29318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:17.443941 29318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:17.444144 29318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:54:17.460808 29318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.133.212:47069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:39189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:54:17.462579 29318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:17.464272 29318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:17.484256 29329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:17.484685 29328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:17.485778 29318 server_base.cc:1034] running on GCE node
W20260925 14:54:17.487614 29331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:17.488291 29318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:17.489349 29318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:17.490557 29318 hybrid_clock.cc:648] HybridClock initialized: now 1790348057490505 us; error 59 us; skew 500 ppm
I20260925 14:54:17.490877 29318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:54:17 dist-test-slave-v5vl krb5kdc[28762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348057, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:17.494900 29318 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:54:17.497255 29318 webserver.cc:460] Webserver started at http://127.17.133.193:33853/ using document root <none> and password file <none>
I20260925 14:54:17.498692 29318 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:17.498843 29318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:17.499226 29318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:54:17 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348057, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:54:17.519388 29318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "7c65b381b7a54ae6832fc9039fcffb6b"
format_stamp: "Formatted at 2026-09-25 14:54:17 on dist-test-slave-v5vl"
server_key: "30f7778c3235b6706295b09d44cfef01"
server_key_iv: "4eac267182a23b1c76767d338559bf03"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:17.520481 29318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "7c65b381b7a54ae6832fc9039fcffb6b"
format_stamp: "Formatted at 2026-09-25 14:54:17 on dist-test-slave-v5vl"
server_key: "30f7778c3235b6706295b09d44cfef01"
server_key_iv: "4eac267182a23b1c76767d338559bf03"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:17.541136 29318 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.009s sys 0.008s
I20260925 14:54:17.577342 29338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:17.586331 29318 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.007s sys 0.008s
I20260925 14:54:17.586637 29318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "7c65b381b7a54ae6832fc9039fcffb6b"
format_stamp: "Formatted at 2026-09-25 14:54:17 on dist-test-slave-v5vl"
server_key: "30f7778c3235b6706295b09d44cfef01"
server_key_iv: "4eac267182a23b1c76767d338559bf03"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:17.586989 29318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:17.636797 29318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:17.642081 29318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:17.642405 29318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:17.643435 29318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:17.644861 29318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:17.644956 29318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:17.645088 29318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:17.645167 29318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:54:17 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348057, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:54:17.830353 29318 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:42243
I20260925 14:54:17.830379 29450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:42243 every 8 connection(s)
I20260925 14:54:17.831952 29318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:17.833803 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 29318
I20260925 14:54:17.834021 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:17.834762 17943 external_mini_cluster.cc:1428] Setting key 1add5da6181f9c5a48bf9ab76ee5c52b
Sep 25 14:54:17 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348057, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:54:17.877949 29453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:17.837818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:56543 (local address 127.17.133.254:39189)
0925 14:54:17.843439 (+ 5621us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:17.843445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:17.844145 (+ 700us) server_negotiation.cc:407] Connection header received
0925 14:54:17.847143 (+ 2998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:17.847151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:17.847254 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:17.847338 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:17.849652 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.850849 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.853276 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.853927 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.854047 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:17.869889 (+ 15842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:17.869899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:17.869907 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:17.869972 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:17.874560 (+ 4588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:17.875387 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:17.875395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:17.875397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:17.875569 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:17.876053 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:17.876061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:17.876064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:17.876477 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:17.876651 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:17.877364 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 14:54:17.877603 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5489,"thread_start_us":5386,"threads_started":1}
I20260925 14:54:17.879086 29454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:17.838366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:39189 (local address 127.17.133.193:56543)
0925 14:54:17.843980 (+ 5614us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:17.844021 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:17.846690 (+ 2669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:17.847893 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:17.847907 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:17.848528 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:17.849445 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:17.849473 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.851082 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.851087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:17.853114 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:17.853129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.855046 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.855053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:17.855392 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:17.856200 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:17.856236 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:17.869541 (+ 13305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:17.874740 (+ 5199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:17.874758 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:17.874776 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:17.875246 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:17.875684 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:17.875689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:17.875692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:17.875914 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:17.876629 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:17.876638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:17.877011 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 14:54:17.877284 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 14:54:17.877808 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5329,"thread_start_us":14162,"threads_started":1}
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:54:17.880926 29451 heartbeater.cc:347] Connected to a master server at 127.17.133.254:39189
I20260925 14:54:17.881263 29451 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:17.882062 29451 heartbeater.cc:511] Master 127.17.133.254:39189 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
I20260925 14:54:17.884404 29238 ts_manager.cc:194] Registered new tserver with Master: 7c65b381b7a54ae6832fc9039fcffb6b (127.17.133.193:42243)
I20260925 14:54:17.885891 29238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:56543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:54:17.914317 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:47069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:39189
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:54:18 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348057, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
W20260925 14:54:18.359961 29457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:18.360255 29457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:18.360427 29457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:18.367771 29457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:18.367883 29457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:18.368090 29457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:54:18.378927 29457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.133.212:47069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:39189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:54:18.380291 29457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:18.381628 29457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:18.394901 29471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:18.394848 29474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:18.394825 29472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:18.395519 29457 server_base.cc:1034] running on GCE node
I20260925 14:54:18.396130 29457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:18.397107 29457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:18.398329 29457 hybrid_clock.cc:648] HybridClock initialized: now 1790348058398297 us; error 46 us; skew 500 ppm
I20260925 14:54:18.398646 29457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:54:18 dist-test-slave-v5vl krb5kdc[28762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348058, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:18.401759 29457 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:54:18.403064 29457 webserver.cc:460] Webserver started at http://127.17.133.194:43627/ using document root <none> and password file <none>
I20260925 14:54:18.403699 29457 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:18.403834 29457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:18.404177 29457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:54:18 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348058, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:54:18.417863 29457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "4d3c9e3ef13b46bc96379747122f1fb2"
format_stamp: "Formatted at 2026-09-25 14:54:18 on dist-test-slave-v5vl"
server_key: "2b790f1610c81ffb2018aaeebb3da0ed"
server_key_iv: "d2ff19f14eb1551eb57fdd8e49b81d1d"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:18.418648 29457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "4d3c9e3ef13b46bc96379747122f1fb2"
format_stamp: "Formatted at 2026-09-25 14:54:18 on dist-test-slave-v5vl"
server_key: "2b790f1610c81ffb2018aaeebb3da0ed"
server_key_iv: "d2ff19f14eb1551eb57fdd8e49b81d1d"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:18.423908 29457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 14:54:18.441094 29481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:18.442633 29457 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260925 14:54:18.442798 29457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "4d3c9e3ef13b46bc96379747122f1fb2"
format_stamp: "Formatted at 2026-09-25 14:54:18 on dist-test-slave-v5vl"
server_key: "2b790f1610c81ffb2018aaeebb3da0ed"
server_key_iv: "d2ff19f14eb1551eb57fdd8e49b81d1d"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:18.443012 29457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:18.468472 29457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:18.472206 29457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:18.472491 29457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:18.473510 29457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:18.474882 29457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:18.474951 29457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:18.475075 29457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:18.475122 29457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:18.525663 29457 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:32969
I20260925 14:54:18.525681 29593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:32969 every 8 connection(s)
I20260925 14:54:18.526963 29457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:18.530578 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 29457
I20260925 14:54:18.530787 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:18.531458 17943 external_mini_cluster.cc:1428] Setting key 0153253c3ae235d10a3280c491178ac7
Sep 25 14:54:18 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348058, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
I20260925 14:54:18.556089 29596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:18.530026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:34701 (local address 127.17.133.254:39189)
0925 14:54:18.532472 (+ 2446us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:18.532478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:18.537361 (+ 4883us) server_negotiation.cc:407] Connection header received
0925 14:54:18.540293 (+ 2932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:18.540299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:18.540384 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:18.540541 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:18.542613 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.543690 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.546260 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.546737 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.546865 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:18.550383 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:18.550388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:18.550391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:18.550419 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:18.553022 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:18.553684 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:18.553691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:18.553699 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:18.553806 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:18.554295 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:18.554298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:18.554301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:18.554615 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:18.554763 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:18.555526 (+ 763us) server_negotiation.cc:299] Negotiation successful
0925 14:54:18.555735 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2333,"thread_start_us":2228,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
I20260925 14:54:18.557267 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:18.530665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:39189 (local address 127.17.133.194:34701)
0925 14:54:18.537202 (+ 6537us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:18.537250 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:18.539893 (+ 2643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:18.540951 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:18.540965 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:18.541612 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:18.542416 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:18.542443 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.543862 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.543870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:18.546099 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:18.546125 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.546961 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.546988 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:18.547237 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:18.547889 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:18.547912 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:18.550163 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:18.553151 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:18.553159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:18.553175 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:18.553565 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:18.553912 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:18.553917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:18.553921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:18.554145 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:18.554738 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:18.554746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:18.555065 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 14:54:18.555414 (+ 349us) client_negotiation.cc:240] Negotiation successful
0925 14:54:18.555904 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6197,"thread_start_us":6002,"threads_started":1}
I20260925 14:54:18.558740 29594 heartbeater.cc:347] Connected to a master server at 127.17.133.254:39189
I20260925 14:54:18.559041 29594 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:18.560113 29594 heartbeater.cc:511] Master 127.17.133.254:39189 requested a full tablet report, sending...
I20260925 14:54:18.561560 29238 ts_manager.cc:194] Registered new tserver with Master: 4d3c9e3ef13b46bc96379747122f1fb2 (127.17.133.194:32969)
I20260925 14:54:18.562304 29238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:34701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:54:18.584020 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:47069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:39189
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:18.892758 29451 heartbeater.cc:503] Master 127.17.133.254:39189 was elected leader, sending a full tablet report...
W20260925 14:54:18.918733 29600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:18.919034 29600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:18.919188 29600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:18.926405 29600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:18.926550 29600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:18.926729 29600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:54:18.936510 29600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.133.212:47069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:39189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:54:18.937851 29600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:18.939170 29600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:18.952746 29606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:18.952780 29608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:18.952693 29605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:18.953573 29600 server_base.cc:1034] running on GCE node
I20260925 14:54:18.954180 29600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:18.955084 29600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:18.956271 29600 hybrid_clock.cc:648] HybridClock initialized: now 1790348058956204 us; error 60 us; skew 500 ppm
I20260925 14:54:18.957016 29600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:54:18 dist-test-slave-v5vl krb5kdc[28762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348058, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:18.960090 29600 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:54:18.961602 29600 webserver.cc:460] Webserver started at http://127.17.133.195:40411/ using document root <none> and password file <none>
I20260925 14:54:18.962265 29600 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:18.962394 29600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:18.962692 29600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:54:18 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348058, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:54:18.977715 29600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "a661741ed8a24b6bbb82d5308013912c"
format_stamp: "Formatted at 2026-09-25 14:54:18 on dist-test-slave-v5vl"
server_key: "7890ed6d519875e742cfc714ff93514f"
server_key_iv: "1412389c08904ede4d12f1cea94bfba5"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:18.978469 29600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "a661741ed8a24b6bbb82d5308013912c"
format_stamp: "Formatted at 2026-09-25 14:54:18 on dist-test-slave-v5vl"
server_key: "7890ed6d519875e742cfc714ff93514f"
server_key_iv: "1412389c08904ede4d12f1cea94bfba5"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:18.983601 29600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 14:54:18.998306 29615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:18.999799 29600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:54:18.999985 29600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "a661741ed8a24b6bbb82d5308013912c"
format_stamp: "Formatted at 2026-09-25 14:54:18 on dist-test-slave-v5vl"
server_key: "7890ed6d519875e742cfc714ff93514f"
server_key_iv: "1412389c08904ede4d12f1cea94bfba5"
server_key_version: "kuduclusterkey@0"
I20260925 14:54:19.000226 29600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:19.018989 29600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:19.022801 29600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:19.023102 29600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:19.024076 29600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:19.025717 29600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:19.025791 29600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:19.025961 29600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:19.026046 29600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:19.062677 29600 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:37851
I20260925 14:54:19.062695 29727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:37851 every 8 connection(s)
I20260925 14:54:19.063930 29600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:19.067337 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 29600
I20260925 14:54:19.067513 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:19.068035 17943 external_mini_cluster.cc:1428] Setting key 52bac7477bb25fcd68e5ed3ed5b97b65
Sep 25 14:54:19 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348058, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:54:19.084152 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.066391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:33375 (local address 127.17.133.254:39189)
0925 14:54:19.066864 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.066868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.067779 (+ 911us) server_negotiation.cc:407] Connection header received
0925 14:54:19.069797 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.069801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.069921 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.070034 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:19.071635 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.072360 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.074332 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.074757 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.074891 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.078170 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:19.078175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:19.078178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:19.078210 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:19.081006 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.081849 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.081856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.081859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.082015 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.082446 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.082450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.082452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.082709 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:19.082872 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.083751 (+ 879us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.083902 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260925 14:54:19.085217 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.066817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:39189 (local address 127.17.133.195:33375)
0925 14:54:19.067618 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.067670 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:19.069536 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:19.070291 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:19.070302 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:19.070890 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:19.071496 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.071512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.072493 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.072497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.074158 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.074196 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.074974 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.074983 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.075207 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.075875 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:19.075897 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:19.077948 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:19.081140 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:19.081146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:19.081160 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:19.081747 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:19.082119 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:19.082123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:19.082126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:19.082332 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:19.082842 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:19.082848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:19.083314 (+ 466us) client_negotiation.cc:769] Sending connection context
0925 14:54:19.083629 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 14:54:19.084010 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":316,"threads_started":1}
I20260925 14:54:19.086535 29728 heartbeater.cc:347] Connected to a master server at 127.17.133.254:39189
I20260925 14:54:19.086843 29728 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:19.087534 29728 heartbeater.cc:511] Master 127.17.133.254:39189 requested a full tablet report, sending...
I20260925 14:54:19.088920 29238 ts_manager.cc:194] Registered new tserver with Master: a661741ed8a24b6bbb82d5308013912c (127.17.133.195:37851)
I20260925 14:54:19.089730 29238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:33375
I20260925 14:54:19.095738 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:54:19.113032 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.100099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.17.133.254:39189)
0925 14:54:19.100288 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.100297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.100722 (+ 425us) server_negotiation.cc:407] Connection header received
0925 14:54:19.100919 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.100926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.101004 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.101119 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:19.102869 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.103641 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.105188 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.105626 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.105759 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.108119 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:19.108126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:19.108131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:19.108177 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:19.110419 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.111053 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.111060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.111063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.111218 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.111603 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.111610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.111613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.111861 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:19.112107 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.112424 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.112657 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:54:19.116640 29238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:19.123612 29238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:19.157182 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.136710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.17.133.193:42243)
0925 14:54:19.149622 (+ 12912us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.149627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.149650 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:54:19.149741 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.149746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.149947 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.150164 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:19.151015 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.152730 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.154554 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.155906 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.156187 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.156369 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.156685 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.156850 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12781,"thread_start_us":12648,"threads_started":1}
I20260925 14:54:19.157191 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.136418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:42243 (local address 127.0.0.1:33290)
0925 14:54:19.147130 (+ 10712us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.147182 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:19.147390 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:19.150186 (+ 2796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:19.150195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:19.150231 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:19.150763 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.150776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.152859 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.152867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.154372 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.154385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.156194 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.156200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.156387 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.156451 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 14:54:19.156629 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 14:54:19.156819 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10393,"thread_start_us":12202,"threads_started":1}
I20260925 14:54:19.160424 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.137323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.17.133.195:37851)
0925 14:54:19.150136 (+ 12813us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.150149 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.150195 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:54:19.150424 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.150436 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.150706 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.150920 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:19.152942 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.154417 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.157198 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.158661 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.158842 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.158986 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.159631 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.159818 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12570,"thread_start_us":12429,"threads_started":1}
I20260925 14:54:19.160475 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.136504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:37851 (local address 127.0.0.1:35092)
0925 14:54:19.148926 (+ 12422us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.148949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:19.149076 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:19.152254 (+ 3178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:19.152258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:19.152293 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:19.152718 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.152731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.154574 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.154579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.157024 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.157038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.159212 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.159218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.159393 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.159450 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 14:54:19.159526 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:54:19.159696 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12281,"mutex_wait_us":97,"thread_start_us":11149,"threads_started":1}
I20260925 14:54:19.160569 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.137232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.17.133.194:32969)
0925 14:54:19.149231 (+ 11999us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.149237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.150997 (+ 1760us) server_negotiation.cc:407] Connection header received
0925 14:54:19.151294 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.151302 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.151538 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.151706 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:19.152683 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.154298 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.157205 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.158654 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.158819 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.158986 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.159662 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.160260 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11821,"thread_start_us":11715,"threads_started":1}
I20260925 14:54:19.161913 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.136974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:32969 (local address 127.0.0.1:38626)
0925 14:54:19.150891 (+ 13917us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.150911 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:19.151098 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:19.152001 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:19.152005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:19.152028 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:19.152482 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.152496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.154464 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.154472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.157024 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.157038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.159158 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.159168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.159378 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.159447 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 14:54:19.159525 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:54:19.160831 (+ 1306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13757,"mutex_wait_us":1260,"thread_start_us":13580,"threads_started":1}
I20260925 14:54:19.161613 29386 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab5506d216c4e148f07a873cca5f199 (DEFAULT_TABLE table=test-table [id=c1022b023c5940e492ccafbdffaca5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:19.163190 29386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab5506d216c4e148f07a873cca5f199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:19.164041 29663 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab5506d216c4e148f07a873cca5f199 (DEFAULT_TABLE table=test-table [id=c1022b023c5940e492ccafbdffaca5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:19.165131 29663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab5506d216c4e148f07a873cca5f199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:19.170404 29529 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab5506d216c4e148f07a873cca5f199 (DEFAULT_TABLE table=test-table [id=c1022b023c5940e492ccafbdffaca5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:19.171583 29529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab5506d216c4e148f07a873cca5f199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:19.183341 29747 tablet_bootstrap.cc:492] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: Bootstrap starting.
I20260925 14:54:19.187500 29748 tablet_bootstrap.cc:492] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: Bootstrap starting.
I20260925 14:54:19.188550 29747 tablet_bootstrap.cc:654] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:19.189935 29747 log.cc:824] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:19.191401 29748 tablet_bootstrap.cc:654] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:19.191663 29749 tablet_bootstrap.cc:492] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: Bootstrap starting.
I20260925 14:54:19.192835 29748 log.cc:824] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:19.194883 29749 tablet_bootstrap.cc:654] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:19.195788 29748 tablet_bootstrap.cc:492] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: No bootstrap required, opened a new log
I20260925 14:54:19.195791 29749 log.cc:824] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:19.196049 29748 ts_tablet_manager.cc:1397] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 14:54:19.197708 29747 tablet_bootstrap.cc:492] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: No bootstrap required, opened a new log
I20260925 14:54:19.197953 29747 ts_tablet_manager.cc:1397] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260925 14:54:19.198431 29749 tablet_bootstrap.cc:492] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: No bootstrap required, opened a new log
I20260925 14:54:19.198660 29749 ts_tablet_manager.cc:1397] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 14:54:19.206373 29749 raft_consensus.cc:348] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } }
I20260925 14:54:19.206741 29749 raft_consensus.cc:374] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:19.206868 29749 raft_consensus.cc:729] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3c9e3ef13b46bc96379747122f1fb2, State: Initialized, Role: FOLLOWER
I20260925 14:54:19.207243 29748 raft_consensus.cc:348] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } }
I20260925 14:54:19.207631 29748 raft_consensus.cc:374] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:19.207482 29749 consensus_queue.cc:260] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } }
I20260925 14:54:19.207722 29748 raft_consensus.cc:729] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a661741ed8a24b6bbb82d5308013912c, State: Initialized, Role: FOLLOWER
I20260925 14:54:19.208298 29748 consensus_queue.cc:260] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } }
I20260925 14:54:19.208724 29594 heartbeater.cc:503] Master 127.17.133.254:39189 was elected leader, sending a full tablet report...
I20260925 14:54:19.208983 29747 raft_consensus.cc:348] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } }
I20260925 14:54:19.209436 29749 ts_tablet_manager.cc:1428] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260925 14:54:19.209461 29747 raft_consensus.cc:374] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:19.209559 29747 raft_consensus.cc:729] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c65b381b7a54ae6832fc9039fcffb6b, State: Initialized, Role: FOLLOWER
I20260925 14:54:19.210232 29747 consensus_queue.cc:260] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } }
I20260925 14:54:19.212080 29747 ts_tablet_manager.cc:1428] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260925 14:54:19.216451 29728 heartbeater.cc:503] Master 127.17.133.254:39189 was elected leader, sending a full tablet report...
I20260925 14:54:19.217212 29748 ts_tablet_manager.cc:1428] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: Time spent starting tablet: real 0.021s user 0.007s sys 0.012s
I20260925 14:54:19.254541 29755 raft_consensus.cc:482] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:19.254809 29755 raft_consensus.cc:504] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } }
I20260925 14:54:19.256886 29755 leader_election.cc:288] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a661741ed8a24b6bbb82d5308013912c (127.17.133.195:37851), 4d3c9e3ef13b46bc96379747122f1fb2 (127.17.133.194:32969)
I20260925 14:54:19.263561 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.257333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:49187 (local address 127.17.133.195:37851)
0925 14:54:19.257540 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.257548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.258230 (+ 682us) server_negotiation.cc:407] Connection header received
0925 14:54:19.258487 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.258494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.258568 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.258632 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:19.259249 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.259988 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.261870 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.262617 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.262782 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.262870 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.263180 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.263309 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:54:19.263598 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.257454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:49459 (local address 127.17.133.194:32969)
0925 14:54:19.257670 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.257676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.258364 (+ 688us) server_negotiation.cc:407] Connection header received
0925 14:54:19.258567 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.258579 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.258668 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.258761 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:19.259369 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.260019 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.261877 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.262636 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.262805 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.262874 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.263251 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.263374 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:54:19.263684 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.257342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:37851 (local address 127.17.133.193:49187)
0925 14:54:19.258091 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.258111 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:19.258275 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:19.258671 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:19.258676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:19.258709 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:19.259100 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.259109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.260121 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.260130 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.261725 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.261736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.262832 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.262839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.262991 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.263050 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 14:54:19.263124 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:54:19.263259 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":40,"thread_start_us":703,"threads_started":1}
I20260925 14:54:19.264288 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.257428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:32969 (local address 127.17.133.193:49459)
0925 14:54:19.258237 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.258260 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:19.258405 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:19.258799 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:19.258803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:19.258840 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:19.259205 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.259216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.260185 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.260188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.261728 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.261739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.262873 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.262880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.263070 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.263092 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:54:19.263203 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:54:19.263315 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":76,"thread_start_us":455,"threads_started":1}
I20260925 14:54:19.265113 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab5506d216c4e148f07a873cca5f199" candidate_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a661741ed8a24b6bbb82d5308013912c" is_pre_election: true
I20260925 14:54:19.265249 29549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab5506d216c4e148f07a873cca5f199" candidate_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" is_pre_election: true
I20260925 14:54:19.265708 29683 raft_consensus.cc:2454] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c65b381b7a54ae6832fc9039fcffb6b in term 0.
I20260925 14:54:19.265707 29549 raft_consensus.cc:2454] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c65b381b7a54ae6832fc9039fcffb6b in term 0.
I20260925 14:54:19.266425 29341 leader_election.cc:302] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c65b381b7a54ae6832fc9039fcffb6b, a661741ed8a24b6bbb82d5308013912c; no voters:
I20260925 14:54:19.266853 29755 raft_consensus.cc:2781] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:19.267002 29755 raft_consensus.cc:482] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:19.267103 29755 raft_consensus.cc:3037] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:19.269287 29755 raft_consensus.cc:504] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } }
I20260925 14:54:19.269865 29755 leader_election.cc:288] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [CANDIDATE]: Term 1 election: Requested vote from peers a661741ed8a24b6bbb82d5308013912c (127.17.133.195:37851), 4d3c9e3ef13b46bc96379747122f1fb2 (127.17.133.194:32969)
I20260925 14:54:19.270300 29549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab5506d216c4e148f07a873cca5f199" candidate_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3c9e3ef13b46bc96379747122f1fb2"
I20260925 14:54:19.270305 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab5506d216c4e148f07a873cca5f199" candidate_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a661741ed8a24b6bbb82d5308013912c"
I20260925 14:54:19.270506 29683 raft_consensus.cc:3037] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:19.270498 29549 raft_consensus.cc:3037] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:19.272624 29549 raft_consensus.cc:2454] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c65b381b7a54ae6832fc9039fcffb6b in term 1.
I20260925 14:54:19.272655 29683 raft_consensus.cc:2454] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c65b381b7a54ae6832fc9039fcffb6b in term 1.
I20260925 14:54:19.273053 29340 leader_election.cc:302] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3c9e3ef13b46bc96379747122f1fb2, 7c65b381b7a54ae6832fc9039fcffb6b; no voters:
I20260925 14:54:19.273351 29755 raft_consensus.cc:2781] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:19.273875 29755 raft_consensus.cc:686] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 1 LEADER]: Becoming Leader. State: Replica: 7c65b381b7a54ae6832fc9039fcffb6b, State: Running, Role: LEADER
I20260925 14:54:19.274365 29755 consensus_queue.cc:237] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } }
W20260925 14:54:19.279341 29595 tablet.cc:2369] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:19.278839 29236 catalog_manager.cc:5495] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c65b381b7a54ae6832fc9039fcffb6b (127.17.133.193). New cstate: current_term: 1 leader_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c65b381b7a54ae6832fc9039fcffb6b" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 42243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 } health_report { overall_health: UNKNOWN } } }
I20260925 14:54:19.315021 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.310032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.17.133.193:42243)
0925 14:54:19.310223 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.310227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.310317 (+ 90us) server_negotiation.cc:407] Connection header received
0925 14:54:19.310531 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.310535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.310617 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.310735 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:19.311035 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.311746 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.313165 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.313557 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.313645 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.314076 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:19.314576 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:19.314681 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.314765 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.314829 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 14:54:19.315891 29729 tablet.cc:2369] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:19.323302 29683 raft_consensus.cc:1261] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [term 1 FOLLOWER]: Refusing update from remote peer 7c65b381b7a54ae6832fc9039fcffb6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:19.323302 29549 raft_consensus.cc:1261] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [term 1 FOLLOWER]: Refusing update from remote peer 7c65b381b7a54ae6832fc9039fcffb6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:19.324332 29758 consensus_queue.cc:1035] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a661741ed8a24b6bbb82d5308013912c" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:19.324769 29755 consensus_queue.cc:1035] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3c9e3ef13b46bc96379747122f1fb2" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:19.342602 29762 mvcc.cc:205] Tried to move back new op lower bound from 7333265650977005568 to 7333265650791182336. Current Snapshot: MvccSnapshot[applied={T|T < 7333265650977005568}]
I20260925 14:54:19.351004 29236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:19.351269 29236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:19.354799 29236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deb9a5c44437462e847b8137e229bc41: Sending DeleteTablet for 3 replicas of tablet 9ab5506d216c4e148f07a873cca5f199
I20260925 14:54:19.355810 29663 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab5506d216c4e148f07a873cca5f199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:19 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:35092
I20260925 14:54:19.355814 29386 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab5506d216c4e148f07a873cca5f199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:19 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:33290
I20260925 14:54:19.356678 29771 tablet_replica.cc:330] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: stopping tablet replica
I20260925 14:54:19.356741 29529 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab5506d216c4e148f07a873cca5f199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:19 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:38626
I20260925 14:54:19.357108 29771 raft_consensus.cc:2229] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:19.357576 29771 raft_consensus.cc:2258] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:19.358544 29772 tablet_replica.cc:330] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: stopping tablet replica
I20260925 14:54:19.358955 29772 raft_consensus.cc:2229] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:19.359072 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 29318
I20260925 14:54:19.361488 29771 ts_tablet_manager.cc:1905] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:19.365154 29772 raft_consensus.cc:2258] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:19.369189 29771 ts_tablet_manager.cc:1918] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:19.369338 29771 log.cc:1196] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/9ab5506d216c4e148f07a873cca5f199
I20260925 14:54:19.369843 29771 ts_tablet_manager.cc:1939] T 9ab5506d216c4e148f07a873cca5f199 P a661741ed8a24b6bbb82d5308013912c: Deleting consensus metadata
I20260925 14:54:19.370146 29773 tablet_replica.cc:330] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: stopping tablet replica
I20260925 14:54:19.370656 29773 raft_consensus.cc:2229] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:19.371238 29773 raft_consensus.cc:2258] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:19.371721 29224 catalog_manager.cc:4841] TS a661741ed8a24b6bbb82d5308013912c (127.17.133.195:37851): tablet 9ab5506d216c4e148f07a873cca5f199 (table test-table [id=c1022b023c5940e492ccafbdffaca5c8]) successfully deleted
I20260925 14:54:19.373225 29772 ts_tablet_manager.cc:1905] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:19.374362 29773 ts_tablet_manager.cc:1905] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:19.381739 29772 ts_tablet_manager.cc:1918] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:19.381740 29773 ts_tablet_manager.cc:1918] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:19.381886 29773 log.cc:1196] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/9ab5506d216c4e148f07a873cca5f199
I20260925 14:54:19.381969 29772 log.cc:1196] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/9ab5506d216c4e148f07a873cca5f199
I20260925 14:54:19.382457 29773 ts_tablet_manager.cc:1939] T 9ab5506d216c4e148f07a873cca5f199 P 4d3c9e3ef13b46bc96379747122f1fb2: Deleting consensus metadata
I20260925 14:54:19.382563 29772 ts_tablet_manager.cc:1939] T 9ab5506d216c4e148f07a873cca5f199 P 7c65b381b7a54ae6832fc9039fcffb6b: Deleting consensus metadata
I20260925 14:54:19.384253 29223 catalog_manager.cc:4841] TS 4d3c9e3ef13b46bc96379747122f1fb2 (127.17.133.194:32969): tablet 9ab5506d216c4e148f07a873cca5f199 (table test-table [id=c1022b023c5940e492ccafbdffaca5c8]) successfully deleted
I20260925 14:54:19.384517 29225 catalog_manager.cc:4841] TS 7c65b381b7a54ae6832fc9039fcffb6b (127.17.133.193:42243): tablet 9ab5506d216c4e148f07a873cca5f199 (table test-table [id=c1022b023c5940e492ccafbdffaca5c8]) successfully deleted
Sep 25 14:54:19 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:54:19.395229 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.360299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.17.133.193:42243)
0925 14:54:19.360443 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.360449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.377652 (+ 17203us) server_negotiation.cc:407] Connection header received
0925 14:54:19.377843 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.377849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.377931 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.378032 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:19.379844 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.381456 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.383518 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.384191 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.384297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.388659 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:19.388674 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:19.388700 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:19.388745 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:19.392528 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.393163 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.393172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.393175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.393348 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.393728 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.393733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.393737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.394163 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:19.394473 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.394616 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.394861 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:54:19.395994 29446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:54:19.495523 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 29318
I20260925 14:54:19.513332 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 29457
Sep 25 14:54:19 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:54:19.528432 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.514006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.17.133.194:32969)
0925 14:54:19.514208 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.514215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.514251 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:54:19.514454 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.514458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.514516 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.514621 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:19.515853 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.516570 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.517908 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.518307 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.518440 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.522517 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:19.522530 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:19.522542 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:19.522577 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:19.525612 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.526296 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.526304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.526306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.526420 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.526791 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.526798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.526801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.527207 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:19.527506 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.527862 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.528082 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:54:19.529292 29589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:54:19.618230 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 29457
I20260925 14:54:19.631131 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 29600
Sep 25 14:54:19 dist-test-slave-v5vl krb5kdc[28762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:54:19.644800 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.631820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.17.133.195:37851)
0925 14:54:19.631976 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.631981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.631998 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:54:19.632107 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.632112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.632182 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.632258 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:19.633536 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.634263 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.635452 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.635833 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.635919 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.639982 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:19.639995 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:19.640011 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:19.640040 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:19.642502 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.643069 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.643075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.643077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.643179 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:19.643483 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:19.643486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:19.643487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:19.643827 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:19.644020 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:19.644302 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 14:54:19.644453 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:54:19.645385 29723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:54:19.732407 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 29600
I20260925 14:54:19.748270 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 29206
I20260925 14:54:19.749471 29268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:54:19.859738 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 29206
I20260925 14:54:19.883075 17943 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:54:20.219444 17943 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:54:20.219632 17943 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:54:20.566320 17943 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:54:20.566 UTC [28792] LOG: received smart shutdown request
2026-09-25 14:54:20.569 UTC [28792] LOG: background worker "logical replication launcher" (PID 28801) exited with exit code 1
2026-09-25 14:54:20.570 UTC [28796] LOG: shutting down
2026-09-25 14:54:20.709 UTC [28792] LOG: database system is shut down
2026-09-25T14:54:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46190 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:20 dist-test-slave-v5vl krb5kdc[29787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:20 dist-test-slave-v5vl krb5kdc[29787](info): set up 2 sockets
Sep 25 14:54:20 dist-test-slave-v5vl krb5kdc[29787](info): commencing operation
krb5kdc: starting...
W20260925 14:54:21.944890 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:54:22 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.133.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.133.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/rangeradmin_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/rangeradmin_127.17.133.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.133.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.133.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/rangerlookup_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/rangerlookup_127.17.133.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.133.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.133.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/HTTP_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/HTTP_127.17.133.212@KRBTEST.COM.keytab.
I20260925 14:54:22.253134 17943 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNtxnNj/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/postgres -l logfile start
2026-09-25 14:54:24.499 UTC [29817] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:54:24.499 UTC [29817] LOG: listening on IPv4 address "127.17.133.212", port 54503
2026-09-25 14:54:24.502 UTC [29817] LOG: listening on Unix socket "/tmp/.s.PGSQL.54503"
2026-09-25 14:54:24.522 UTC [29820] LOG: database system was shut down at 2026-09-25 14:54:23 UTC
2026-09-25 14:54:24.525 UTC [29817] LOG: database system is ready to accept connections
I20260925 14:54:25.484467 17943 mini_postgres.cc:96] Postgres bound to 54503
2026-09-25 14:54:25.505 UTC [29828] FATAL: database "slave" does not exist
127.17.133.212:54503 - accepting connections
I20260925 14:54:25.505641 17943 mini_ranger.cc:161] Starting Ranger...
I20260925 14:54:25.529321 17943 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:54:25.950443 17943 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:54:25.950567 17943 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-admin
2026-09-25 14:54:26,218 [I] DB FLAVOR :POSTGRES
2026-09-25 14:54:26,218 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:54:26,218 [I] Checking connection..
2026-09-25 14:54:26,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:54:26,475 [I] Checking connection passed.
2026-09-25 14:54:26,476 [I] --------- Verifying version history table ---------
2026-09-25 14:54:26,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:54:26,713 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:54:26,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:54:26,954 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:54:26,955 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:54:26,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:54:27.200 UTC [29892] WARNING: there is no transaction in progress
2026-09-25 14:54:27,206 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:54:27,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:54:27,450 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:54:27,451 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:54:27,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:54:27,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:54:27,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v5vl.c.gcp-upstream.internal','N') ;"
2026-09-25 14:54:28,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:54:28,419 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:54:28,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:54:28,654 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:54:28,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:54:28,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:54:29,158 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:54:29,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:54:30.101 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.116 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.132 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.147 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.163 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.181 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.234 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.240 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.253 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.262 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.272 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.286 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.295 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.301 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.308 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30.314 UTC [30000] WARNING: there is no transaction in progress
2026-09-25 14:54:30,938 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:54:30,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:54:31,188 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:54:31,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:54:31,450 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:54:31,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:54:31,694 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:54:31,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:54:31,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:54:32,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v5vl.c.gcp-upstream.internal';"
2026-09-25 14:54:32,444 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:54:32,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:54:32,692 [I] DB_PATCHES have already been applied
I20260925 14:54:32.698714 17943 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNtxnNj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:54:32.698791 17943 mini_ranger.cc:193] Using host: 127.17.133.212
I20260925 14:54:32.712745 17943 mini_ranger.cc:227] Ranger admin URL: http://127.17.133.212:43413
Sep 25, 2026 2:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:54:33.818184 17943 mini_ranger.cc:160] Time spent starting Ranger: real 8.313s user 0.002s sys 0.032s
Sep 25, 2026 2:54:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:54:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:54:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:54:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:54:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:54:54.161115 17943 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.133.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.133.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/rangerkms_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.133.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/rangerkms_127.17.133.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.133.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.133.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.133.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/HTTP_127.17.133.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.133.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/HTTP_127.17.133.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:54:54 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348094, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:54:54.347082 17943 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:54:54.370189 17943 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:54:55.134526 17943 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:54:55.134653 17943 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-kms
I20260925 14:54:55.134696 17943 mini_ranger_kms.cc:227] Using postgres at 127.17.133.212:54503
2026-09-25 14:54:55,471 [I] DB FLAVOR :POSTGRES
2026-09-25 14:54:55,472 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:54:55,472 [I] Checking connection
2026-09-25 14:54:55,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:54:55,707 [I] connection success
2026-09-25 14:54:55,708 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:54:55,708 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:54:55,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:54:55,927 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:54:55,927 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:54:55,927 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:54:55,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.133.212:54503/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:54:56,178 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:54:56.603150 17943 mini_ranger_kms.cc:310] Created kms service
I20260925 14:54:56.788144 17943 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:54:56.830981 17943 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:54:56 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348094, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:54:57.234812 17943 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:54:57.235194 17943 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNtxnNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNtxnNj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:54:57.235260 17943 mini_ranger_kms.cc:240] Using host: 127.17.133.212
I20260925 14:54:57.253528 17943 mini_ranger_kms.cc:276] Ranger KMS PID: 30180
I20260925 14:54:57.253759 17943 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.133.212:50051
Sep 25, 2026 2:54:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:54:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:54:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:54:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:54:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:54:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 14:54:59 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348099, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.133.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:59.529342 17943 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.182s user 0.068s sys 0.281s
I20260925 14:54:59.529611 17943 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:54:59.529795 17943 mini_ranger_kms.cc:401] 127.17.133.212:50051/kms/v1/keys
Sep 25 14:54:59 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348099, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.133.212@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
Sep 25 14:55:00 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348099, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.133.212@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
Sep 25 14:55:02 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:55:02.491686 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:42123
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:42123
--ranger_config_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:50051/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:02.803383 30231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:02.803730 30231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:02.803874 30231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:02.810890 30231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:02.811028 30231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:02.811120 30231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:02.811223 30231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:02.811300 30231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:02.821204 30231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.133.212:50051/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:42123
--ranger_config_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:02.822546 30231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:02.823870 30231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:02.835116 30236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:02.835188 30237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:02.835204 30239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:02.835975 30231 server_base.cc:1034] running on GCE node
I20260925 14:55:02.836771 30231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:02.838081 30231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:02.839262 30231 hybrid_clock.cc:648] HybridClock initialized: now 1790348102839214 us; error 48 us; skew 500 ppm
I20260925 14:55:02.839572 30231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:02 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348102, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:02.842387 30231 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:55:02.843743 30231 webserver.cc:460] Webserver started at http://127.17.133.254:44531/ using document root <none> and password file <none>
I20260925 14:55:02.844348 30231 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:02.844484 30231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:02.844830 30231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:02 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348102, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:02.902696 30231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "4a7be70ac72341d09d5346c6979a3b60"
format_stamp: "Formatted at 2026-09-25 14:55:02 on dist-test-slave-v5vl"
server_key: "e634cadbf22d36b75230f47952852af9"
server_key_iv: "6afb313802d0d15f8533743e53176051"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:02.903860 30231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "4a7be70ac72341d09d5346c6979a3b60"
format_stamp: "Formatted at 2026-09-25 14:55:02 on dist-test-slave-v5vl"
server_key: "e634cadbf22d36b75230f47952852af9"
server_key_iv: "6afb313802d0d15f8533743e53176051"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:02.910887 30231 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260925 14:55:02.930406 30246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:02.932134 30231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:55:02.932324 30231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "4a7be70ac72341d09d5346c6979a3b60"
format_stamp: "Formatted at 2026-09-25 14:55:02 on dist-test-slave-v5vl"
server_key: "e634cadbf22d36b75230f47952852af9"
server_key_iv: "6afb313802d0d15f8533743e53176051"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:02.932638 30231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:02.960750 30231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:02.965489 30231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:02.965797 30231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:02.985491 30231 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:42123
I20260925 14:55:02.985486 30297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:42123 every 8 connection(s)
I20260925 14:55:02.987174 30231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:02.990919 30298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:02.994786 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 30231
I20260925 14:55:02.995007 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:55:02.995839 17943 external_mini_cluster.cc:1428] Setting key cc1ee0f1d8071c9d781ade5378af00d3
I20260925 14:55:03.006181 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Bootstrap starting.
I20260925 14:55:03.011821 30298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:03.013675 30298 log.cc:824] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:03.016883 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: No bootstrap required, opened a new log
I20260925 14:55:03.026757 30298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } }
I20260925 14:55:03.027320 30298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:03.027472 30298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7be70ac72341d09d5346c6979a3b60, State: Initialized, Role: FOLLOWER
Sep 25 14:55:03 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:03.028158 30298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } }
I20260925 14:55:03.028393 30298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:03.028537 30298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:03.028810 30298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:03.031916 30298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } }
I20260925 14:55:03.032487 30298 leader_election.cc:302] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a7be70ac72341d09d5346c6979a3b60; no voters:
I20260925 14:55:03.033211 30298 leader_election.cc:288] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:03.033344 30303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:03.039268 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:02.997917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.17.133.254:42123)
0925 14:55:03.013113 (+ 15196us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:03.013132 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:03.013184 (+ 52us) server_negotiation.cc:407] Connection header received
0925 14:55:03.015797 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:03.015829 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:03.016345 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:03.016840 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:03.018560 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:03.020744 (+ 2184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:03.022684 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:03.023815 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:03.024011 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:03.029150 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:03.029161 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:03.029182 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:03.029229 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:03.033400 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:03.034321 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:03.034330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:03.034339 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:03.034529 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:03.034972 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:03.034976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:03.034980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:03.035697 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:03.037109 (+ 1412us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:03.037428 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 14:55:03.037876 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14924,"thread_start_us":14641,"threads_started":1}
I20260925 14:55:03.040470 30303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 1 LEADER]: Becoming Leader. State: Replica: 4a7be70ac72341d09d5346c6979a3b60, State: Running, Role: LEADER
I20260925 14:55:03.041180 30303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } }
I20260925 14:55:03.043469 30298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:03.044868 30305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a7be70ac72341d09d5346c6979a3b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } } }
I20260925 14:55:03.045413 30305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:03.046020 30304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a7be70ac72341d09d5346c6979a3b60. Latest consensus state: current_term: 1 leader_uuid: "4a7be70ac72341d09d5346c6979a3b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } } }
I20260925 14:55:03.046414 30304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:03.052882 30308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:03.059602 30308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:03.079434 30308 catalog_manager.cc:1347] Generated new cluster ID: 0379072357c6400ca6e4aab267c7e583
I20260925 14:55:03.079560 30308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:03.089939 30298 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:55:03.092698 30308 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:03.099741 30308 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:03.116480 30308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Generated new TSK 0
I20260925 14:55:03.117743 30308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:55:03.870019 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:50051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42123
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:04 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348104, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 14:55:04.281170 30343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:04.281533 30343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:04.281755 30343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:04.293071 30343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:04.293259 30343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:04.293469 30343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:55:04.308944 30343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.133.212:50051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:04.310772 30343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:04.312448 30343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:04.329305 30353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:04.330708 30354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:04.339448 30343 server_base.cc:1034] running on GCE node
W20260925 14:55:04.339982 30356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:04.340772 30343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:04.341964 30343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:04.343251 30343 hybrid_clock.cc:648] HybridClock initialized: now 1790348104343183 us; error 55 us; skew 500 ppm
I20260925 14:55:04.343652 30343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:04 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348104, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:04.347740 30343 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:55:04.349650 30343 webserver.cc:460] Webserver started at http://127.17.133.193:35023/ using document root <none> and password file <none>
I20260925 14:55:04.350611 30343 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:04.350791 30343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:04.351186 30343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:04 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348104, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:04.376310 30343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "01744dc81d6d40b5bbdf3ed4146991c5"
format_stamp: "Formatted at 2026-09-25 14:55:04 on dist-test-slave-v5vl"
server_key: "c7cf2b59fa4dbbad6196e8f909464689"
server_key_iv: "1d6c12e6104b11ec2f5eea4913b3512b"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:04.377740 30343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "01744dc81d6d40b5bbdf3ed4146991c5"
format_stamp: "Formatted at 2026-09-25 14:55:04 on dist-test-slave-v5vl"
server_key: "c7cf2b59fa4dbbad6196e8f909464689"
server_key_iv: "1d6c12e6104b11ec2f5eea4913b3512b"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:04.383539 30343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:55:04.408872 30363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:04.410668 30343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 14:55:04.410926 30343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "01744dc81d6d40b5bbdf3ed4146991c5"
format_stamp: "Formatted at 2026-09-25 14:55:04 on dist-test-slave-v5vl"
server_key: "c7cf2b59fa4dbbad6196e8f909464689"
server_key_iv: "1d6c12e6104b11ec2f5eea4913b3512b"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:04.411454 30343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:04.453174 30343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:04.458551 30343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:04.458900 30343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:04.463395 30343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:04.465289 30343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:04.465387 30343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:04.465559 30343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:04.465626 30343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:04.520296 30475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:36585 every 8 connection(s)
I20260925 14:55:04.520258 30343 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:36585
I20260925 14:55:04.522243 30343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:04.531284 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 30343
I20260925 14:55:04.531482 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:04.532208 17943 external_mini_cluster.cc:1428] Setting key ede50173d06791874bbcc2d3236c6ca3
Sep 25 14:55:04 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348104, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
I20260925 14:55:04.580495 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:04.524823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:34673 (local address 127.17.133.254:42123)
0925 14:55:04.543583 (+ 18760us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:04.543592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:04.559453 (+ 15861us) server_negotiation.cc:407] Connection header received
0925 14:55:04.561879 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:04.561884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:04.561976 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:04.562082 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:04.564050 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:04.565132 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:04.567380 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:04.568007 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:04.568104 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:04.573077 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:04.573084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:04.573089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:04.573141 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:04.576801 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:04.577720 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:04.577726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:04.577730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:04.577866 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:04.578371 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:04.578376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:04.578380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:04.578730 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:04.578931 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:04.579885 (+ 954us) server_negotiation.cc:299] Negotiation successful
0925 14:55:04.580120 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18547,"thread_start_us":18775,"threads_started":1}
I20260925 14:55:04.582970 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:04.525585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42123 (local address 127.17.133.193:34673)
0925 14:55:04.558890 (+ 33305us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:04.558949 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:04.561413 (+ 2464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:04.562503 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:04.562515 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:04.563086 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:04.563869 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:04.563892 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:04.565317 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:04.565322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:04.567210 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:04.567224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:04.568433 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:04.568439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:04.568726 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:04.569504 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:04.569532 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:04.572789 (+ 3257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:04.576960 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:04.576969 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:04.576989 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:04.577579 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:04.578002 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:04.578006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:04.578009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:04.578245 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:04.579041 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:04.579049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:04.579467 (+ 418us) client_negotiation.cc:769] Sending connection context
0925 14:55:04.581033 (+ 1566us) client_negotiation.cc:240] Negotiation successful
0925 14:55:04.581572 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32985,"thread_start_us":32717,"threads_started":1}
I20260925 14:55:04.585295 30476 heartbeater.cc:347] Connected to a master server at 127.17.133.254:42123
I20260925 14:55:04.585632 30476 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:04.586470 30476 heartbeater.cc:511] Master 127.17.133.254:42123 requested a full tablet report, sending...
I20260925 14:55:04.588881 30263 ts_manager.cc:194] Registered new tserver with Master: 01744dc81d6d40b5bbdf3ed4146991c5 (127.17.133.193:36585)
I20260925 14:55:04.590806 30263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:34673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:55:04.617609 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:50051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42123
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:04 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348104, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
Sep 25 14:55:04 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348104, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
W20260925 14:55:05.103045 30482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:05.103421 30482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:05.103628 30482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:05.114702 30482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:05.114861 30482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:05.115052 30482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:55:05.130401 30482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.133.212:50051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:05.132179 30482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:05.133949 30482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:05.154449 30489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:05.155450 30490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:05.161789 30482 server_base.cc:1034] running on GCE node
W20260925 14:55:05.162070 30492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:05.162875 30482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:05.176864 30482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:05.178208 30482 hybrid_clock.cc:648] HybridClock initialized: now 1790348105178136 us; error 70 us; skew 500 ppm
I20260925 14:55:05.178686 30482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:05 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348105, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:05.182984 30482 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:55:05.197706 30482 webserver.cc:460] Webserver started at http://127.17.133.194:38285/ using document root <none> and password file <none>
I20260925 14:55:05.198697 30482 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:05.198864 30482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:05.199265 30482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:05 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348105, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:05.223529 30482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c"
format_stamp: "Formatted at 2026-09-25 14:55:05 on dist-test-slave-v5vl"
server_key: "dc0b18c65b473d1fd4e139ba03b18a53"
server_key_iv: "be87366331c7de41b5d49cc6c41248f9"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:05.225628 30482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c"
format_stamp: "Formatted at 2026-09-25 14:55:05 on dist-test-slave-v5vl"
server_key: "dc0b18c65b473d1fd4e139ba03b18a53"
server_key_iv: "be87366331c7de41b5d49cc6c41248f9"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:05.243677 30482 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.012s
I20260925 14:55:05.266511 30501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:05.268548 30482 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260925 14:55:05.268887 30482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c"
format_stamp: "Formatted at 2026-09-25 14:55:05 on dist-test-slave-v5vl"
server_key: "dc0b18c65b473d1fd4e139ba03b18a53"
server_key_iv: "be87366331c7de41b5d49cc6c41248f9"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:05.269254 30482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:05.296432 30482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:05.301709 30482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:05.302083 30482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:05.303272 30482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:05.305111 30482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:05.305254 30482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:05.305420 30482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:05.305523 30482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:05.342535 30482 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:39943
I20260925 14:55:05.342581 30618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:39943 every 8 connection(s)
I20260925 14:55:05.344290 30482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:05.348860 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 30482
I20260925 14:55:05.349090 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:05.349694 17943 external_mini_cluster.cc:1428] Setting key f62132ec716d1735fecb1390299ba079
Sep 25 14:55:05 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348105, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:05.383090 30621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:05.347826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42123 (local address 127.17.133.194:33377)
0925 14:55:05.353332 (+ 5506us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:05.353395 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:05.356762 (+ 3367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:05.362917 (+ 6155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:05.362931 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:05.363612 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:05.364420 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:05.364446 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.366312 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.366317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:05.368483 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:05.368499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.369684 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.369690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:05.369973 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:05.370807 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:05.370836 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:05.373663 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:05.378584 (+ 4921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:05.378594 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:05.378623 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:05.379171 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:05.379603 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:05.379608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:05.379611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:05.379839 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:05.380528 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:05.380536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:05.380982 (+ 446us) client_negotiation.cc:769] Sending connection context
0925 14:55:05.381265 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 14:55:05.381760 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5107,"thread_start_us":4843,"threads_started":1}
I20260925 14:55:05.384140 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:05.348852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:33377 (local address 127.17.133.254:42123)
0925 14:55:05.361068 (+ 12216us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:05.361078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:05.361104 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:55:05.361181 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:05.361189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:05.361357 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:05.361515 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:05.364946 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.366106 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.368667 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.369270 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.369361 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:05.373965 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:05.373973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:05.373978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:05.374019 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:05.378412 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:05.379318 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:05.379324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:05.379327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:05.379474 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:05.379976 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:05.379981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:05.379985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:05.380318 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:05.380443 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:05.383564 (+ 3121us) server_negotiation.cc:299] Negotiation successful
0925 14:55:05.383778 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12047,"thread_start_us":11918,"threads_started":1}
I20260925 14:55:05.386173 30619 heartbeater.cc:347] Connected to a master server at 127.17.133.254:42123
I20260925 14:55:05.386494 30619 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:05.387396 30619 heartbeater.cc:511] Master 127.17.133.254:42123 requested a full tablet report, sending...
I20260925 14:55:05.389257 30263 ts_manager.cc:194] Registered new tserver with Master: 1e0d68b6118b43618fa4aaa7e1f57e2c (127.17.133.194:39943)
I20260925 14:55:05.390365 30263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:33377
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:55:05.426309 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:50051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42123
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:05.594494 30476 heartbeater.cc:503] Master 127.17.133.254:42123 was elected leader, sending a full tablet report...
W20260925 14:55:05.810359 30625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:05.810662 30625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:05.810751 30625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:05.818186 30625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:05.818305 30625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:05.818416 30625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:55:05.828179 30625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.133.212:50051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:05.829483 30625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:05.830729 30625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:05.843282 30633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:05.843345 30630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:05.843323 30631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:05.844151 30625 server_base.cc:1034] running on GCE node
I20260925 14:55:05.844823 30625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:05.845624 30625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:05.846841 30625 hybrid_clock.cc:648] HybridClock initialized: now 1790348105846794 us; error 60 us; skew 500 ppm
I20260925 14:55:05.847139 30625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:05 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348105, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:05.849936 30625 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:55:05.851248 30625 webserver.cc:460] Webserver started at http://127.17.133.195:44155/ using document root <none> and password file <none>
I20260925 14:55:05.852021 30625 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:05.852160 30625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:05.852451 30625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:05 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348105, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:05.870385 30625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "0cf974bcdcee4f95a1d41f306499b183"
format_stamp: "Formatted at 2026-09-25 14:55:05 on dist-test-slave-v5vl"
server_key: "d86d9f638eb206b6755dfc5626249854"
server_key_iv: "bf54363ab7f17a660211d513165618fe"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:05.871503 30625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "0cf974bcdcee4f95a1d41f306499b183"
format_stamp: "Formatted at 2026-09-25 14:55:05 on dist-test-slave-v5vl"
server_key: "d86d9f638eb206b6755dfc5626249854"
server_key_iv: "bf54363ab7f17a660211d513165618fe"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:05.877797 30625 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 14:55:05.892695 30640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:05.894116 30625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:55:05.894285 30625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "0cf974bcdcee4f95a1d41f306499b183"
format_stamp: "Formatted at 2026-09-25 14:55:05 on dist-test-slave-v5vl"
server_key: "d86d9f638eb206b6755dfc5626249854"
server_key_iv: "bf54363ab7f17a660211d513165618fe"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:05.894495 30625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:05.914362 30625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:05.918601 30625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:05.918897 30625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:05.919808 30625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:05.921146 30625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:05.921221 30625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:05.921392 30625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:05.921442 30625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:05.952553 30625 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:44171
I20260925 14:55:05.952565 30752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:44171 every 8 connection(s)
I20260925 14:55:05.953930 30625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:05.963969 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 30625
I20260925 14:55:05.964170 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:05.964720 17943 external_mini_cluster.cc:1428] Setting key f247b549a4982c9c5f77d67c0c0eb27e
Sep 25 14:55:05 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348105, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:05.972517 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:05.956314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:59919 (local address 127.17.133.254:42123)
0925 14:55:05.956840 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:05.956847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:05.957696 (+ 849us) server_negotiation.cc:407] Connection header received
0925 14:55:05.959369 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:05.959373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:05.959452 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:05.959546 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:05.961040 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.961760 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.963211 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.963721 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.963847 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:05.967593 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:05.967599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:05.967604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:05.967644 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:05.970021 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:05.970693 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:05.970697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:05.970700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:05.970815 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:05.971190 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:05.971193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:05.971195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:05.971429 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:05.971541 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:05.972109 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 14:55:05.972261 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":291,"threads_started":1}
I20260925 14:55:05.973510 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:05.956835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42123 (local address 127.17.133.195:59919)
0925 14:55:05.957534 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:05.957571 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:05.959134 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:05.959781 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:05.959789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:05.960243 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:05.960885 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:05.960904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.961887 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.961890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:05.963091 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:05.963101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.963929 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.963933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:05.964191 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:05.965019 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:05.965053 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:05.967338 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:05.970160 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:05.970166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:05.970178 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:05.970582 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:05.970899 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:05.970903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:05.970905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:05.971088 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:05.971524 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:05.971529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:05.971814 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 14:55:05.972019 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 14:55:05.972381 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":266,"threads_started":1}
I20260925 14:55:05.975445 30753 heartbeater.cc:347] Connected to a master server at 127.17.133.254:42123
I20260925 14:55:05.975760 30753 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:05.976675 30753 heartbeater.cc:511] Master 127.17.133.254:42123 requested a full tablet report, sending...
I20260925 14:55:05.978266 30263 ts_manager.cc:194] Registered new tserver with Master: 0cf974bcdcee4f95a1d41f306499b183 (127.17.133.195:44171)
I20260925 14:55:05.979187 30263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:59919
I20260925 14:55:05.979910 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:05.981567 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 30343
Sep 25 14:55:05 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:55:05.997232 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:05.982526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.17.133.193:36585)
0925 14:55:05.983151 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:05.983164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:05.983187 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:55:05.983240 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:05.983245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:05.983414 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:05.983525 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:05.984939 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.985908 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.987343 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:05.987865 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:05.988009 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:05.992136 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:05.992145 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:05.992157 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:05.992211 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:05.994814 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:05.995447 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:05.995452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:05.995454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:05.995585 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:05.995883 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:05.995886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:05.995888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:05.996173 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:05.996406 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:05.996693 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 14:55:05.996874 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":316,"threads_started":1}
I20260925 14:55:05.998492 30471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:06.080441 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 30343
I20260925 14:55:06.096063 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 30482
Sep 25 14:55:06 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:55:06.116737 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:06.096877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.17.133.194:39943)
0925 14:55:06.097392 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:06.097399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:06.097423 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:55:06.097496 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:06.097503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:06.097725 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:06.097860 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:06.099103 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.100085 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.101398 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.101919 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.102010 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:06.111731 (+ 9721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:06.111741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:06.111754 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:06.111787 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:06.114348 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:06.114973 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:06.114979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:06.114981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:06.115105 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:06.115399 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:06.115402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:06.115404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:06.115705 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:06.115897 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:06.116192 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 14:55:06.116393 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":276,"threads_started":1}
I20260925 14:55:06.117906 30614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:06.202039 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 30482
I20260925 14:55:06.217710 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 30625
Sep 25 14:55:06 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:55:06.233879 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:06.218700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.17.133.195:44171)
0925 14:55:06.219350 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:06.219357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:06.219384 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:55:06.219438 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:06.219445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:06.219646 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:06.219810 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:06.221201 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.222211 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.223601 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.224170 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.224314 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:06.228659 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:06.228670 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:06.228686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:06.228716 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:06.231287 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:06.231916 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:06.231922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:06.231924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:06.232036 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:06.232361 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:06.232368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:06.232370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:06.232776 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:06.233060 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:06.233339 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 14:55:06.233569 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":362,"threads_started":1}
I20260925 14:55:06.235011 30748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:06.317075 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 30625
I20260925 14:55:06.332378 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 30231
I20260925 14:55:06.333637 30293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:06.438614 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 30231
I20260925 14:55:06.459782 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:42123
--webserver_interface=127.17.133.254
--webserver_port=44531
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:42123
--ranger_config_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:50051/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.133.212:50051/kms with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:06.772921 30765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:06.773247 30765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:06.773417 30765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:06.780783 30765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:06.780936 30765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:06.781057 30765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:06.781162 30765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:06.781241 30765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:06.790799 30765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.133.212:50051/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:42123
--ranger_config_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=44531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:06.792129 30765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:06.793453 30765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:06.804953 30771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:06.804931 30773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:06.805017 30770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:06.805473 30765 server_base.cc:1034] running on GCE node
I20260925 14:55:06.806243 30765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:06.807514 30765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:06.808744 30765 hybrid_clock.cc:648] HybridClock initialized: now 1790348106808705 us; error 44 us; skew 500 ppm
I20260925 14:55:06.809091 30765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:06 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348106, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:06.811982 30765 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:55:06.813336 30765 webserver.cc:460] Webserver started at http://127.17.133.254:44531/ using document root <none> and password file <none>
I20260925 14:55:06.813957 30765 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:06.814093 30765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:55:06 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348106, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:06.860908 30765 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 14:55:06.863628 30781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:06.864954 30765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:06.865145 30765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "4a7be70ac72341d09d5346c6979a3b60"
format_stamp: "Formatted at 2026-09-25 14:55:02 on dist-test-slave-v5vl"
server_key: "e634cadbf22d36b75230f47952852af9"
server_key_iv: "6afb313802d0d15f8533743e53176051"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:06.866060 30765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:06.900701 30765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:06.904171 30765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:06.904449 30765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:06.922127 30765 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:42123
I20260925 14:55:06.922147 30832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:42123 every 8 connection(s)
I20260925 14:55:06.923586 30765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:06.930148 30833 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:55:06.932503 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 30765
I20260925 14:55:06.933321 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:50051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:36585
--local_ip_for_outbound_sockets=127.17.133.193
--tserver_master_addrs=127.17.133.254:42123
--webserver_port=35023
--webserver_interface=127.17.133.193
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.133.212:50051/kms with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:06.933399 30833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Bootstrap starting.
I20260925 14:55:06.951215 30833 log.cc:824] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:06.959573 30833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:55:06.960098 30833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Bootstrap complete.
I20260925 14:55:06.968977 30833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } }
I20260925 14:55:06.969928 30833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7be70ac72341d09d5346c6979a3b60, State: Initialized, Role: FOLLOWER
I20260925 14:55:06.970578 30833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } }
I20260925 14:55:06.970870 30833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:06.971012 30833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:06.971201 30833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:55:06.974944 30833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } }
I20260925 14:55:06.975425 30833 leader_election.cc:302] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a7be70ac72341d09d5346c6979a3b60; no voters:
I20260925 14:55:06.976059 30833 leader_election.cc:288] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:55:06.976195 30837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:55:06.977352 30837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [term 2 LEADER]: Becoming Leader. State: Replica: 4a7be70ac72341d09d5346c6979a3b60, State: Running, Role: LEADER
I20260925 14:55:06.977977 30837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } }
I20260925 14:55:06.978294 30833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:06.984818 30839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a7be70ac72341d09d5346c6979a3b60. Latest consensus state: current_term: 2 leader_uuid: "4a7be70ac72341d09d5346c6979a3b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } } }
I20260925 14:55:06.985337 30839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:06.984798 30838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4a7be70ac72341d09d5346c6979a3b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7be70ac72341d09d5346c6979a3b60" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 42123 } } }
I20260925 14:55:06.985550 30838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:06.987767 30843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:06.993567 30843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:06.994289 30833 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:55:06.994928 30843 catalog_manager.cc:1259] Loaded cluster ID: 0379072357c6400ca6e4aab267c7e583
I20260925 14:55:06.995009 30843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:06.999009 30843 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:07.000409 30843 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Loaded TSK: 0
I20260925 14:55:07.001961 30843 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:55:07.359100 30835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:07.359556 30835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:07.359762 30835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:07.371066 30835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:07.371285 30835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:07.371536 30835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:55:07.385915 30835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.133.212:50051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:36585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=35023
--tserver_master_addrs=127.17.133.254:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:07.387427 30835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:07.389068 30835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:07.435894 30873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:07.436157 30870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:07.436908 30871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:07.441931 30835 server_base.cc:1034] running on GCE node
I20260925 14:55:07.442595 30835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:07.443666 30835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:07.445514 30835 hybrid_clock.cc:648] HybridClock initialized: now 1790348107445438 us; error 58 us; skew 500 ppm
I20260925 14:55:07.445938 30835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:07 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:07.452291 30835 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:55:07.501653 30835 webserver.cc:460] Webserver started at http://127.17.133.193:35023/ using document root <none> and password file <none>
I20260925 14:55:07.502780 30835 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:07.503026 30835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:55:07 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:07.559665 30835 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.006s sys 0.005s
I20260925 14:55:07.578819 30882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:07.581584 30835 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.007s sys 0.008s
I20260925 14:55:07.581889 30835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "01744dc81d6d40b5bbdf3ed4146991c5"
format_stamp: "Formatted at 2026-09-25 14:55:04 on dist-test-slave-v5vl"
server_key: "c7cf2b59fa4dbbad6196e8f909464689"
server_key_iv: "1d6c12e6104b11ec2f5eea4913b3512b"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:07.585444 30835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:07.629087 30835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:07.633373 30835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:07.633634 30835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:07.634696 30835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:07.636242 30835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:07.636358 30835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:07.636485 30835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:07.636623 30835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:07.717484 30835 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:36585
I20260925 14:55:07.717597 30994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:36585 every 8 connection(s)
I20260925 14:55:07.719430 30835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:07.722476 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 30835
I20260925 14:55:07.723263 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:50051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:39943
--local_ip_for_outbound_sockets=127.17.133.194
--tserver_master_addrs=127.17.133.254:42123
--webserver_port=38285
--webserver_interface=127.17.133.194
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.133.212:50051/kms with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:07 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:07.781279 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:07.730215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42123 (local address 127.17.133.193:54411)
0925 14:55:07.749441 (+ 19226us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:07.749525 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:07.753162 (+ 3637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:07.761019 (+ 7857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:07.761032 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:07.761720 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:07.762530 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:07.762556 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.765112 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.765117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:07.767043 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:07.767056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.768897 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.768902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:07.769158 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:07.769908 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:07.769936 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:07.772522 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:07.776749 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:07.776758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:07.776774 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:07.777222 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:07.777689 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:07.777701 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:07.777703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:07.777878 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:07.778871 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:07.778882 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:07.779177 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 14:55:07.779426 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 14:55:07.779983 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18769,"thread_start_us":18471,"threads_started":1}
I20260925 14:55:07.781283 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:07.731576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:54411 (local address 127.17.133.254:42123)
0925 14:55:07.738381 (+ 6805us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:07.738402 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:07.749886 (+ 11484us) server_negotiation.cc:407] Connection header received
0925 14:55:07.754009 (+ 4123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:07.754033 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:07.754621 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:07.755069 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:07.762808 (+ 7739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.764881 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.767201 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.768223 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.768388 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:07.772808 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:07.772817 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:07.772853 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:07.772895 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:07.776548 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.777387 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.777395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.777402 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.777567 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.778012 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.778017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.778020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.778644 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:07.778864 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:07.779624 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 14:55:07.779969 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6503,"thread_start_us":6257,"threads_started":1}
I20260925 14:55:07.784243 30995 heartbeater.cc:347] Connected to a master server at 127.17.133.254:42123
I20260925 14:55:07.784538 30995 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:07.785246 30995 heartbeater.cc:511] Master 127.17.133.254:42123 requested a full tablet report, sending...
W20260925 14:55:07.787891 30995 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:42123 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:55:07.788146 30995 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:07.788417 30995 heartbeater.cc:511] Master 127.17.133.254:42123 requested a full tablet report, sending...
I20260925 14:55:07.789521 30995 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:07.789772 30995 heartbeater.cc:511] Master 127.17.133.254:42123 requested a full tablet report, sending...
W20260925 14:55:07.790773 30995 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:55:07.887130 31006 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 14:55:08.116777 30999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:08.117069 30999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:08.117233 30999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:08.125137 30999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:08.125263 30999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:08.125450 30999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:55:08.140790 30999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.133.212:50051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:39943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=38285
--tserver_master_addrs=127.17.133.254:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:08.142650 30999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:08.144362 30999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:08.192714 31016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.193583 31015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.196883 31018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:08.202790 30999 server_base.cc:1034] running on GCE node
I20260925 14:55:08.203703 30999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:08.206907 30999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:08.208256 30999 hybrid_clock.cc:648] HybridClock initialized: now 1790348108208169 us; error 70 us; skew 500 ppm
I20260925 14:55:08.208738 30999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:08 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:08.212975 30999 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:55:08.216359 30999 webserver.cc:460] Webserver started at http://127.17.133.194:38285/ using document root <none> and password file <none>
I20260925 14:55:08.217447 30999 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:08.217662 30999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:55:08 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:08.257201 30999 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.005s sys 0.007s
I20260925 14:55:08.260254 31026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Sep 25 14:55:08 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:08.261873 30999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:08.262053 30999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c"
format_stamp: "Formatted at 2026-09-25 14:55:05 on dist-test-slave-v5vl"
server_key: "dc0b18c65b473d1fd4e139ba03b18a53"
server_key_iv: "be87366331c7de41b5d49cc6c41248f9"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:08.263067 30999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:08.285985 30999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:08.290836 30999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:08.291225 30999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:08.298413 30999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:08.299978 30999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:08.300096 30999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.300249 30999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:08.300334 30999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.369650 30999 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:39943
I20260925 14:55:08.369774 31139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:39943 every 8 connection(s)
I20260925 14:55:08.371513 30999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:08.377769 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 30999
I20260925 14:55:08.378499 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.133.212:50051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:44171
--local_ip_for_outbound_sockets=127.17.133.195
--tserver_master_addrs=127.17.133.254:42123
--webserver_port=44155
--webserver_interface=127.17.133.195
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.133.212:50051/kms with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:08 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:08.399739 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:08.375203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42123 (local address 127.17.133.194:59871)
0925 14:55:08.376009 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:08.376057 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:08.378694 (+ 2637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:08.379689 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:08.379703 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:08.380366 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:08.381209 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:08.381244 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.382899 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.382908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:08.384999 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:08.385016 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.387267 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.387278 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:08.387774 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:08.388540 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:08.388569 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:08.390817 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:08.394970 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:08.394980 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:08.394999 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:08.395430 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:08.395877 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:08.395882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:08.395886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:08.396148 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:08.396903 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:08.396914 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:08.397375 (+ 461us) client_negotiation.cc:769] Sending connection context
0925 14:55:08.397685 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 14:55:08.398266 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":758,"threads_started":1}
I20260925 14:55:08.401724 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:08.374476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:59871 (local address 127.17.133.254:42123)
0925 14:55:08.374999 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:08.375005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:08.376185 (+ 1180us) server_negotiation.cc:407] Connection header received
0925 14:55:08.379085 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:08.379091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:08.379187 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:08.379308 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:08.381432 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.382676 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.386004 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.386727 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.386846 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:08.391036 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:08.391042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:08.391047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:08.391088 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:08.394847 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:08.395586 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:08.395593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:08.395599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:08.395743 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:08.396282 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:08.396289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:08.396295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:08.396724 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:08.400829 (+ 4105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:08.400972 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 14:55:08.401220 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":246,"threads_started":1}
I20260925 14:55:08.403767 31140 heartbeater.cc:347] Connected to a master server at 127.17.133.254:42123
I20260925 14:55:08.404107 31140 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:08.405118 31140 heartbeater.cc:511] Master 127.17.133.254:42123 requested a full tablet report, sending...
I20260925 14:55:08.407439 30798 ts_manager.cc:194] Registered new tserver with Master: 1e0d68b6118b43618fa4aaa7e1f57e2c (127.17.133.194:39943)
I20260925 14:55:08.409876 30798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:59871
Sep 25 14:55:08 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:08.790937 30995 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:08.791383 30995 heartbeater.cc:511] Master 127.17.133.254:42123 requested a full tablet report, sending...
I20260925 14:55:08.792959 30798 ts_manager.cc:194] Registered new tserver with Master: 01744dc81d6d40b5bbdf3ed4146991c5 (127.17.133.193:36585)
I20260925 14:55:08.794602 30798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:54411
W20260925 14:55:08.823467 31144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:08.823972 31144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:08.824184 31144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:08.834683 31144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:08.834873 31144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:08.835076 31144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:55:08.849731 31144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.133.212:50051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:44171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=44155
--tserver_master_addrs=127.17.133.254:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:08.851563 31144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:08.853386 31144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:08.876358 31149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.877189 31150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.881662 31152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:08.883661 31144 server_base.cc:1034] running on GCE node
I20260925 14:55:08.884438 31144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:08.892827 31144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:08.900389 31144 hybrid_clock.cc:648] HybridClock initialized: now 1790348108900262 us; error 120 us; skew 500 ppm
I20260925 14:55:08.900935 31144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:08 dist-test-slave-v5vl krb5kdc[29787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:08.905336 31144 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:55:08.926312 31144 webserver.cc:460] Webserver started at http://127.17.133.195:44155/ using document root <none> and password file <none>
I20260925 14:55:08.927656 31144 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:08.927906 31144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:55:08 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:08.969651 31144 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.009s sys 0.004s
I20260925 14:55:08.979355 31160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.981297 31144 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260925 14:55:08.981525 31144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "0cf974bcdcee4f95a1d41f306499b183"
format_stamp: "Formatted at 2026-09-25 14:55:05 on dist-test-slave-v5vl"
server_key: "d86d9f638eb206b6755dfc5626249854"
server_key_iv: "bf54363ab7f17a660211d513165618fe"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:08.983002 31144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:09.029313 31144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:09.034752 31144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:09.035121 31144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:09.042601 31144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:09.050690 31144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:09.050983 31144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:09.051144 31144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:09.051256 31144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:55:09 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for HTTP/127.17.133.212@KRBTEST.COM
I20260925 14:55:09.364449 31144 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:44171
I20260925 14:55:09.364530 31273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:44171 every 8 connection(s)
I20260925 14:55:09.366945 31144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:09.369359 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 31144
I20260925 14:55:09.414757 31140 heartbeater.cc:503] Master 127.17.133.254:42123 was elected leader, sending a full tablet report...
I20260925 14:55:09.435940 31276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.370451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.17.133.254:42123)
0925 14:55:09.383152 (+ 12701us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.383159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.383191 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:55:09.383255 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.383260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.383349 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.383460 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:09.385565 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.386702 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.389009 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.389790 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.389903 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.418696 (+ 28793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:09.418703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:09.418708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:09.418757 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:09.423597 (+ 4840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.427923 (+ 4326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.427929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.427933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.428089 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.429279 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.429284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.429288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.429640 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:09.435464 (+ 5824us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.435547 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.435711 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12580,"thread_start_us":12444,"threads_started":1}
Sep 25 14:55:09 dist-test-slave-v5vl krb5kdc[29787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:09.456528 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.396790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:38587 (local address 127.17.133.254:42123)
0925 14:55:09.435338 (+ 38548us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.435342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.435357 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:55:09.438554 (+ 3197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.438562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.438654 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.438967 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:09.441318 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.442494 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.444960 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.445637 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.445773 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.450823 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:09.450830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:09.450836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:09.450875 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:09.453576 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.454294 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.454302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.454310 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.454423 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.454855 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.454859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.454863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.455118 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:09.455268 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.455953 (+ 685us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.456151 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38421,"thread_start_us":39285,"threads_started":1}
I20260925 14:55:09.457559 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.397078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:42123 (local address 127.17.133.195:38587)
0925 14:55:09.431654 (+ 34576us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.431723 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.437167 (+ 5444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.439286 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.439299 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.439992 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:09.441022 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.441051 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.442681 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.442687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.444788 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.444802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.445908 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.445915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.446195 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.447014 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:09.447051 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:09.450483 (+ 3432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:09.453726 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:09.453735 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:09.453752 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:09.454160 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:09.454531 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:09.454537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:09.454540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:09.454740 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:09.455243 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:09.455251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:09.455599 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.455861 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.456395 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34197,"thread_start_us":36889,"threads_started":1}
I20260925 14:55:09.459012 31274 heartbeater.cc:347] Connected to a master server at 127.17.133.254:42123
I20260925 14:55:09.459309 31274 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:09.459990 31274 heartbeater.cc:511] Master 127.17.133.254:42123 requested a full tablet report, sending...
I20260925 14:55:09.461481 30798 ts_manager.cc:194] Registered new tserver with Master: 0cf974bcdcee4f95a1d41f306499b183 (127.17.133.195:44171)
I20260925 14:55:09.462641 30798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:38587
I20260925 14:55:09.463826 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:09.481405 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.466963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.17.133.254:42123)
0925 14:55:09.467149 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.467156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.467695 (+ 539us) server_negotiation.cc:407] Connection header received
0925 14:55:09.467878 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.467882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.467943 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.468014 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:09.469789 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.470495 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.472397 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.472903 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.472994 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.476139 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:09.476143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:09.476147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:09.476175 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:09.478680 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.479580 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.479584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.479586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.479707 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.480107 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.480110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.480111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.480350 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:09.480529 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.480969 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.481131 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:55:09.486948 30798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:09.498310 30798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:09.544039 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.514939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.17.133.194:39943)
0925 14:55:09.527702 (+ 12763us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.527716 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.531777 (+ 4061us) server_negotiation.cc:407] Connection header received
0925 14:55:09.532253 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.532262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.532476 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.532666 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.536699 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.538272 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.540947 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.542382 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.542536 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.542703 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.543423 (+ 720us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.543615 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12511,"thread_start_us":12308,"threads_started":1}
I20260925 14:55:09.544917 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.513545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36585 (local address 127.0.0.1:60652)
0925 14:55:09.531156 (+ 17611us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.531194 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.531379 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.532499 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.532510 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.532556 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.533713 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.533725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.535715 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.535721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.542006 (+ 6285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.542019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.544072 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.544078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.544249 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.544280 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.544352 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.544519 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17357,"mutex_wait_us":647,"thread_start_us":17232,"threads_started":1}
I20260925 14:55:09.545749 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.514070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.17.133.193:36585)
0925 14:55:09.530533 (+ 16463us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.530547 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.531594 (+ 1047us) server_negotiation.cc:407] Connection header received
0925 14:55:09.531722 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.531731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.532012 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.532196 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.533903 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.535536 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.542168 (+ 6632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.543569 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.543752 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.543922 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.545243 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.545424 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16249,"thread_start_us":15943,"threads_started":1}
I20260925 14:55:09.545231 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.514300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:39943 (local address 127.0.0.1:52268)
0925 14:55:09.531657 (+ 17357us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.531675 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.532048 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.532957 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.532962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.532990 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.533492 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.533506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.538457 (+ 4951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.538462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.540724 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.540745 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.542893 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.542898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.543101 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.543189 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.544661 (+ 1472us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.544794 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17244,"mutex_wait_us":4,"thread_start_us":17485,"threads_started":1}
I20260925 14:55:09.549808 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.533187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:44171 (local address 127.0.0.1:46138)
0925 14:55:09.541081 (+ 7894us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.541098 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.541211 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.542918 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.542923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.542946 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.543375 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.543386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.545221 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.545226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.547123 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.547140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.549112 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.549118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.549267 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.549294 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.549376 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.549487 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7766,"mutex_wait_us":63,"thread_start_us":7544,"threads_started":1}
I20260925 14:55:09.549922 31074 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3913b3ec8c454fa749b66da83e5109 (DEFAULT_TABLE table=test-table [id=51f73fda84a94d9981743f9ec4aab672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:09.549921 30930 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3913b3ec8c454fa749b66da83e5109 (DEFAULT_TABLE table=test-table [id=51f73fda84a94d9981743f9ec4aab672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:09.551280 30930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3913b3ec8c454fa749b66da83e5109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:09.551297 31074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3913b3ec8c454fa749b66da83e5109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:09.551635 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.532060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.17.133.195:44171)
0925 14:55:09.537541 (+ 5481us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.537555 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.541205 (+ 3650us) server_negotiation.cc:407] Connection header received
0925 14:55:09.541360 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.541367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.541556 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.541733 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.543581 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.545069 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.547303 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.548645 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.548830 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.548982 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.550105 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.550253 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5341,"thread_start_us":5220,"threads_started":1}
I20260925 14:55:09.555151 31209 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3913b3ec8c454fa749b66da83e5109 (DEFAULT_TABLE table=test-table [id=51f73fda84a94d9981743f9ec4aab672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:09.556520 31209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3913b3ec8c454fa749b66da83e5109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:09.572518 31303 tablet_bootstrap.cc:492] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: Bootstrap starting.
I20260925 14:55:09.572896 31302 tablet_bootstrap.cc:492] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5: Bootstrap starting.
I20260925 14:55:09.577003 31302 tablet_bootstrap.cc:654] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:09.577980 31303 tablet_bootstrap.cc:654] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:09.578526 31302 log.cc:824] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:09.578661 31304 tablet_bootstrap.cc:492] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: Bootstrap starting.
I20260925 14:55:09.579250 31303 log.cc:824] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:09.583254 31304 tablet_bootstrap.cc:654] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:09.584478 31304 log.cc:824] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:09.588277 31302 tablet_bootstrap.cc:492] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5: No bootstrap required, opened a new log
I20260925 14:55:09.588480 31303 tablet_bootstrap.cc:492] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: No bootstrap required, opened a new log
I20260925 14:55:09.588644 31302 ts_tablet_manager.cc:1397] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260925 14:55:09.588805 31303 ts_tablet_manager.cc:1397] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260925 14:55:09.599455 31304 tablet_bootstrap.cc:492] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: No bootstrap required, opened a new log
I20260925 14:55:09.599311 31303 raft_consensus.cc:348] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } }
I20260925 14:55:09.599795 31304 ts_tablet_manager.cc:1397] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.012s
I20260925 14:55:09.599985 31303 raft_consensus.cc:374] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:09.600142 31303 raft_consensus.cc:729] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0d68b6118b43618fa4aaa7e1f57e2c, State: Initialized, Role: FOLLOWER
I20260925 14:55:09.600237 31302 raft_consensus.cc:348] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } }
I20260925 14:55:09.600755 31302 raft_consensus.cc:374] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:09.600886 31302 raft_consensus.cc:729] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01744dc81d6d40b5bbdf3ed4146991c5, State: Initialized, Role: FOLLOWER
I20260925 14:55:09.601011 31303 consensus_queue.cc:260] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } }
I20260925 14:55:09.601707 31302 consensus_queue.cc:260] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } }
I20260925 14:55:09.609386 31303 ts_tablet_manager.cc:1428] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20260925 14:55:09.609778 31302 ts_tablet_manager.cc:1428] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5: Time spent starting tablet: real 0.021s user 0.004s sys 0.014s
I20260925 14:55:09.610145 30995 heartbeater.cc:503] Master 127.17.133.254:42123 was elected leader, sending a full tablet report...
I20260925 14:55:09.612630 31304 raft_consensus.cc:348] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } }
I20260925 14:55:09.613065 31304 raft_consensus.cc:374] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:09.613160 31304 raft_consensus.cc:729] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf974bcdcee4f95a1d41f306499b183, State: Initialized, Role: FOLLOWER
I20260925 14:55:09.613744 31304 consensus_queue.cc:260] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } }
I20260925 14:55:09.616650 31274 heartbeater.cc:503] Master 127.17.133.254:42123 was elected leader, sending a full tablet report...
I20260925 14:55:09.616959 31304 ts_tablet_manager.cc:1428] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260925 14:55:09.639673 31141 tablet.cc:2369] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:09.689284 31277 tablet.cc:2369] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:09.730176 31309 raft_consensus.cc:482] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:09.730500 31309 raft_consensus.cc:504] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } }
I20260925 14:55:09.732606 31309 leader_election.cc:288] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf974bcdcee4f95a1d41f306499b183 (127.17.133.195:44171), 1e0d68b6118b43618fa4aaa7e1f57e2c (127.17.133.194:39943)
I20260925 14:55:09.740490 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.733137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:45325 (local address 127.17.133.194:39943)
0925 14:55:09.733403 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.733409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.734050 (+ 641us) server_negotiation.cc:407] Connection header received
0925 14:55:09.734365 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.734373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.734449 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.734530 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.735210 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.736284 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.738729 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.739490 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.739662 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.739737 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.740122 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.740254 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 14:55:09.740559 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.733072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:39943 (local address 127.17.133.193:45325)
0925 14:55:09.733939 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.733957 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.734177 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.734566 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.734571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.734596 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.735030 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.735044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.736432 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.736437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.738526 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.738539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.739690 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.739696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.739881 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.739926 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.740065 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.740185 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":62,"thread_start_us":509,"threads_started":1}
I20260925 14:55:09.740705 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.733066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:42695 (local address 127.17.133.195:44171)
0925 14:55:09.733264 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.733272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.734420 (+ 1148us) server_negotiation.cc:407] Connection header received
0925 14:55:09.734556 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.734561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.734624 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.734709 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.735329 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.736013 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.738627 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.739767 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.739938 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.740004 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.740332 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.740451 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:55:09.741351 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.733641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:44171 (local address 127.17.133.193:42695)
0925 14:55:09.734308 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.734324 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.734435 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.734727 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.734733 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.734757 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.735173 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.735183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.736142 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.736149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.738456 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.738470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.739987 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.739996 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.740156 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.740184 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.740874 (+ 690us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.741001 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":44,"thread_start_us":363,"threads_started":1}
I20260925 14:55:09.741899 31094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3913b3ec8c454fa749b66da83e5109" candidate_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" is_pre_election: true
I20260925 14:55:09.742241 31229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3913b3ec8c454fa749b66da83e5109" candidate_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf974bcdcee4f95a1d41f306499b183" is_pre_election: true
I20260925 14:55:09.742457 31094 raft_consensus.cc:2454] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01744dc81d6d40b5bbdf3ed4146991c5 in term 0.
I20260925 14:55:09.742774 31229 raft_consensus.cc:2454] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01744dc81d6d40b5bbdf3ed4146991c5 in term 0.
I20260925 14:55:09.743243 30885 leader_election.cc:302] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01744dc81d6d40b5bbdf3ed4146991c5, 1e0d68b6118b43618fa4aaa7e1f57e2c; no voters:
I20260925 14:55:09.743873 31309 raft_consensus.cc:2781] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:09.744010 31309 raft_consensus.cc:482] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:09.744095 31309 raft_consensus.cc:3037] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:09.747177 31309 raft_consensus.cc:504] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } }
I20260925 14:55:09.748214 31229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3913b3ec8c454fa749b66da83e5109" candidate_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf974bcdcee4f95a1d41f306499b183"
I20260925 14:55:09.748279 31094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3913b3ec8c454fa749b66da83e5109" candidate_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c"
I20260925 14:55:09.748411 31229 raft_consensus.cc:3037] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:09.748466 31094 raft_consensus.cc:3037] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:09.750684 31229 raft_consensus.cc:2454] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01744dc81d6d40b5bbdf3ed4146991c5 in term 1.
I20260925 14:55:09.750876 31309 leader_election.cc:288] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf974bcdcee4f95a1d41f306499b183 (127.17.133.195:44171), 1e0d68b6118b43618fa4aaa7e1f57e2c (127.17.133.194:39943)
I20260925 14:55:09.751094 30884 leader_election.cc:302] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01744dc81d6d40b5bbdf3ed4146991c5, 0cf974bcdcee4f95a1d41f306499b183; no voters:
I20260925 14:55:09.751396 31309 raft_consensus.cc:2781] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:09.751545 31094 raft_consensus.cc:2454] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01744dc81d6d40b5bbdf3ed4146991c5 in term 1.
I20260925 14:55:09.751964 31309 raft_consensus.cc:686] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [term 1 LEADER]: Becoming Leader. State: Replica: 01744dc81d6d40b5bbdf3ed4146991c5, State: Running, Role: LEADER
I20260925 14:55:09.752764 31309 consensus_queue.cc:237] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } }
I20260925 14:55:09.757510 30798 catalog_manager.cc:5495] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01744dc81d6d40b5bbdf3ed4146991c5 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01744dc81d6d40b5bbdf3ed4146991c5" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 } health_report { overall_health: UNKNOWN } } }
W20260925 14:55:09.758627 30996 tablet.cc:2369] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:09.818928 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.813193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.17.133.193:36585)
0925 14:55:09.813410 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.813417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.813438 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:55:09.813543 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.813548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.813659 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.813836 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:09.814156 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.815274 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.816491 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.817168 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.817287 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.817743 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:09.818379 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:09.818517 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.818562 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.818638 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:55:09.827561 31094 raft_consensus.cc:1261] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [term 1 FOLLOWER]: Refusing update from remote peer 01744dc81d6d40b5bbdf3ed4146991c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:09.827657 31229 raft_consensus.cc:1261] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [term 1 FOLLOWER]: Refusing update from remote peer 01744dc81d6d40b5bbdf3ed4146991c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:09.828560 31313 consensus_queue.cc:1035] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf974bcdcee4f95a1d41f306499b183" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:09.828936 31309 consensus_queue.cc:1035] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e0d68b6118b43618fa4aaa7e1f57e2c" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 39943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:09.869823 30797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:09.870111 30797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:09.874465 30797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a7be70ac72341d09d5346c6979a3b60: Sending DeleteTablet for 3 replicas of tablet 8d3913b3ec8c454fa749b66da83e5109
I20260925 14:55:09.875980 30930 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3913b3ec8c454fa749b66da83e5109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:09 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:60652
I20260925 14:55:09.876227 31074 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3913b3ec8c454fa749b66da83e5109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:09 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:52268
I20260925 14:55:09.876965 31209 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3913b3ec8c454fa749b66da83e5109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:09 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:46138
I20260925 14:55:09.882979 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 30835
I20260925 14:55:09.909677 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.883769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.17.133.193:36585)
0925 14:55:09.883983 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.883992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.884042 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:55:09.884238 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.884245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.884364 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.884482 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:09.889526 (+ 5044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.891027 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.893986 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.894903 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.895055 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.899989 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:09.900008 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:09.900033 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:09.900092 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:09.904912 (+ 4820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.906058 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.906071 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.906078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.906365 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.906918 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.906926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.906931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.907744 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:09.908128 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.908834 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.909150 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:55:09.911294 30990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:09.915858 31327 tablet_replica.cc:330] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: stopping tablet replica
I20260925 14:55:09.916330 31327 raft_consensus.cc:2229] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:09.917004 31327 raft_consensus.cc:2258] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:09.917770 31328 tablet_replica.cc:330] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: stopping tablet replica
I20260925 14:55:09.921094 31328 raft_consensus.cc:2229] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:09.921109 31327 ts_tablet_manager.cc:1905] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:09.924396 31328 raft_consensus.cc:2258] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:09.946884 31328 ts_tablet_manager.cc:1905] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:09.954988 31327 ts_tablet_manager.cc:1918] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:09.955261 31327 log.cc:1196] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/8d3913b3ec8c454fa749b66da83e5109
I20260925 14:55:09.955834 31327 ts_tablet_manager.cc:1939] T 8d3913b3ec8c454fa749b66da83e5109 P 1e0d68b6118b43618fa4aaa7e1f57e2c: Deleting consensus metadata
I20260925 14:55:09.957885 30784 catalog_manager.cc:4841] TS 1e0d68b6118b43618fa4aaa7e1f57e2c (127.17.133.194:39943): tablet 8d3913b3ec8c454fa749b66da83e5109 (table test-table [id=51f73fda84a94d9981743f9ec4aab672]) successfully deleted
I20260925 14:55:09.959882 31328 ts_tablet_manager.cc:1918] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:09.960137 31328 log.cc:1196] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/8d3913b3ec8c454fa749b66da83e5109
I20260925 14:55:09.960844 31328 ts_tablet_manager.cc:1939] T 8d3913b3ec8c454fa749b66da83e5109 P 0cf974bcdcee4f95a1d41f306499b183: Deleting consensus metadata
I20260925 14:55:09.962908 30783 catalog_manager.cc:4841] TS 0cf974bcdcee4f95a1d41f306499b183 (127.17.133.195:44171): tablet 8d3913b3ec8c454fa749b66da83e5109 (table test-table [id=51f73fda84a94d9981743f9ec4aab672]) successfully deleted
I20260925 14:55:10.047830 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 30835
I20260925 14:55:10.047828 31326 tablet_replica.cc:330] T 8d3913b3ec8c454fa749b66da83e5109 P 01744dc81d6d40b5bbdf3ed4146991c5: stopping tablet replica
W20260925 14:55:10.063764 30783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.133.193:36585 (error 108)
I20260925 14:55:10.064522 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 30999
I20260925 14:55:10.065989 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.065116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36585 (local address 127.0.0.1:60674)
0925 14:55:10.065674 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:10.065830 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":356,"threads_started":1}
W20260925 14:55:10.066296 30783 catalog_manager.cc:4584] TS 01744dc81d6d40b5bbdf3ed4146991c5 (127.17.133.193:36585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d3913b3ec8c454fa749b66da83e5109: Network error: Client connection negotiation failed: client connection to 127.17.133.193:36585: connect: Connection refused (error 111)
I20260925 14:55:10.077917 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.065297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.17.133.194:39943)
0925 14:55:10.065483 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:10.065488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:10.065506 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:55:10.065676 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:10.065681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:10.065763 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:10.065865 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:10.067426 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.068152 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.069453 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.069998 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.070135 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:10.072561 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:10.072575 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:10.072614 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:10.072651 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:10.075088 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.075734 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.075742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.075746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.075897 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.076296 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.076303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.076306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.076765 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:10.077076 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:10.077351 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 14:55:10.077571 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:55:10.078537 31135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:10.119921 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.119501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36585 (local address 127.0.0.1:60682)
0925 14:55:10.119693 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:10.119787 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 14:55:10.121037 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.120668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36585 (local address 127.0.0.1:60696)
0925 14:55:10.120849 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:10.120912 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 14:55:10.166872 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 30999
I20260925 14:55:10.181682 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.181091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36585 (local address 127.0.0.1:60710)
0925 14:55:10.181362 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:10.181479 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 14:55:10.182780 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.182343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36585 (local address 127.0.0.1:60722)
0925 14:55:10.182515 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:10.182610 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 14:55:10.186249 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 31144
I20260925 14:55:10.199546 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.186959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.17.133.195:44171)
0925 14:55:10.187196 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:10.187203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:10.187222 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:55:10.187377 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:10.187381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:10.187442 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:10.187556 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:10.188939 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.189632 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.190871 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.191261 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.191384 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:10.194090 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:10.194103 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:10.194119 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:10.194172 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:10.196796 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.197519 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.197526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.197528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.197653 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.198029 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.198035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.198039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.198392 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:10.198700 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:10.199000 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 14:55:10.199210 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 14:55:10.200197 31269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:10.263387 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.262804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36585 (local address 127.0.0.1:60730)
0925 14:55:10.263143 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:10.263231 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 14:55:10.264508 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.264118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36585 (local address 127.0.0.1:60738)
0925 14:55:10.264305 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:10.264390 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 14:55:10.284664 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 31144
I20260925 14:55:10.301672 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 30765
I20260925 14:55:10.302706 30828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:10.411973 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 30765
I20260925 14:55:10.435389 17943 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:55:10.780196 17943 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:55:10.780422 17943 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:55:11.125057 17943 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:55:11.125 UTC [29817] LOG: received smart shutdown request
2026-09-25 14:55:11.128 UTC [29817] LOG: background worker "logical replication launcher" (PID 29826) exited with exit code 1
2026-09-25 14:55:11.128 UTC [29821] LOG: shutting down
2026-09-25 14:55:11.275 UTC [29817] LOG: database system is shut down
2026-09-25T14:55:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50541 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:11 dist-test-slave-v5vl krb5kdc[31343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:11 dist-test-slave-v5vl krb5kdc[31343](info): set up 2 sockets
Sep 25 14:55:11 dist-test-slave-v5vl krb5kdc[31343](info): commencing operation
krb5kdc: starting...
W20260925 14:55:12.472772 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:55:12 dist-test-slave-v5vl krb5kdc[31343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:55:12.677870 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:41415
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:41415,127.17.133.253:35483,127.17.133.252:37755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:12.989868 31356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:12.990140 31356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:12.990260 31356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:12.997417 31356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:12.997530 31356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:12.997622 31356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:12.997726 31356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:12.997803 31356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:13.008307 31356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:41415,127.17.133.253:35483,127.17.133.252:37755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:13.009593 31356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:13.010850 31356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:13.022395 31361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.022356 31364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.022327 31362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:13.023456 31356 server_base.cc:1034] running on GCE node
I20260925 14:55:13.024205 31356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:13.025620 31356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:13.026839 31356 hybrid_clock.cc:648] HybridClock initialized: now 1790348113026782 us; error 47 us; skew 500 ppm
I20260925 14:55:13.027173 31356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:13 dist-test-slave-v5vl krb5kdc[31343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.030263 31356 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:55:13.031612 31356 webserver.cc:460] Webserver started at http://127.17.133.254:38355/ using document root <none> and password file <none>
I20260925 14:55:13.032269 31356 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:13.032405 31356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:13.032759 31356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:13.035630 31356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "eca71ce1312c46b8abffd4db1b63c6eb"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-v5vl"
server_key: "ce785a405dd5bb929a51a114a861ed50"
server_key_iv: "00348954d55333b629325b2d18e387a6"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.036401 31356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "eca71ce1312c46b8abffd4db1b63c6eb"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-v5vl"
server_key: "ce785a405dd5bb929a51a114a861ed50"
server_key_iv: "00348954d55333b629325b2d18e387a6"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.041241 31356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 14:55:13.044384 31371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.045660 31356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:55:13.045825 31356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "eca71ce1312c46b8abffd4db1b63c6eb"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-v5vl"
server_key: "ce785a405dd5bb929a51a114a861ed50"
server_key_iv: "00348954d55333b629325b2d18e387a6"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.046011 31356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:13.071658 31356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:13.075915 31356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:13.076210 31356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:13.094897 31356 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:41415
I20260925 14:55:13.094898 31422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:41415 every 8 connection(s)
I20260925 14:55:13.096341 31356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:13.097158 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 31356
I20260925 14:55:13.097424 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:55:13.098150 17943 external_mini_cluster.cc:1428] Setting key e452706a77ff91b8b07b8b3e824bc77a
I20260925 14:55:13.100443 31423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.108876 31423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.133.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.253@KRBTEST.COM" created.
Sep 25 14:55:13 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:13.134330 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.112879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41415 (local address 127.0.0.1:57084)
0925 14:55:13.113839 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.113906 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.116833 (+ 2927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.117757 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.117770 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.118596 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.119135 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.119162 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.120940 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.120944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.122276 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.122288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.123353 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.123363 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.123598 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.124315 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.124350 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.126933 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.130092 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.130104 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.130117 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.130494 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.130905 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.130910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.130913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.131067 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.131810 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.131817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.132049 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.132300 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.132930 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":144,"thread_start_us":416,"threads_started":1}
I20260925 14:55:13.134364 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.112879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.17.133.254:41415)
0925 14:55:13.114805 (+ 1926us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.114810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.114848 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:55:13.117244 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.117249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.117486 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.117649 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.119440 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.120779 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.122421 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.123098 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.123230 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.127152 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.127159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.127171 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.127198 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.129957 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.130625 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.130633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.130639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.130785 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.131168 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.131176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.131180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.131674 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.131883 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.132414 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.132821 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":1735,"spinlock_wait_cycles":24320,"thread_start_us":1513,"threads_started":1}
I20260925 14:55:13.137619 31423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:55:13.138789 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.138391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48256)
0925 14:55:13.138563 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.138674 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 14:55:13.139103 31374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
I20260925 14:55:13.140828 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.140343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48268)
0925 14:55:13.140526 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.140650 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 14:55:13.141124 31423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:35483: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab.
Entry for principal kudu/127.17.133.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab.
I20260925 14:55:13.151625 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.253:35483
--webserver_interface=127.17.133.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab
--principal=kudu/127.17.133.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:41415,127.17.133.253:35483,127.17.133.252:37755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:13.190495 31423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } attempt: 1
I20260925 14:55:13.191797 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.191227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48274)
0925 14:55:13.191452 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.191623 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 14:55:13.192836 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.192367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48284)
0925 14:55:13.192538 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.192662 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 14:55:13.193089 31423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:35483: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
I20260925 14:55:13.261363 31423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } attempt: 2
I20260925 14:55:13.262651 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.262096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48296)
0925 14:55:13.262357 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.262468 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 14:55:13.263669 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.263271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48304)
0925 14:55:13.263434 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.263514 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 14:55:13.263877 31423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:35483: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
I20260925 14:55:13.355149 31423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } attempt: 3
I20260925 14:55:13.356367 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.355856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48318)
0925 14:55:13.356088 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.356209 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 14:55:13.357422 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.357011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48328)
0925 14:55:13.357160 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.357240 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 14:55:13.357635 31423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:35483: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
W20260925 14:55:13.465472 31430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:13.465747 31430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:13.465834 31430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:13.472731 31430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:13.472822 31430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:13.472865 31430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:13.472920 31430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:13.472961 31430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:13.482380 31430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:41415,127.17.133.253:35483,127.17.133.252:37755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab
--principal=kudu/127.17.133.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.253:35483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.133.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:13.483588 31430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:13.484802 31430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:13.496392 31435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.496419 31436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.496464 31438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:13.497346 31430 server_base.cc:1034] running on GCE node
I20260925 14:55:13.498118 31430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:13.499580 31430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:13.500808 31430 hybrid_clock.cc:648] HybridClock initialized: now 1790348113500755 us; error 48 us; skew 500 ppm
I20260925 14:55:13.500871 31423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } attempt: 4
I20260925 14:55:13.501216 31430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:55:13.502066 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.501453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48330)
0925 14:55:13.501771 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.501883 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260925 14:55:13.503327 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.502904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48344)
0925 14:55:13.503074 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.503176 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 14:55:13.503537 31423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:35483: Network error: Client connection negotiation failed: client connection to 127.17.133.253:35483: connect: Connection refused (error 111)
Sep 25 14:55:13 dist-test-slave-v5vl krb5kdc[31343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.504714 31430 init.cc:377] Logged in from keytab as kudu/127.17.133.253@KRBTEST.COM (short username kudu)
I20260925 14:55:13.506107 31430 webserver.cc:460] Webserver started at http://127.17.133.253:43189/ using document root <none> and password file <none>
I20260925 14:55:13.506778 31430 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:13.506904 31430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:13.507212 31430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:13.510121 31430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/data/instance:
uuid: "5dfaaf818e8447dfb4175ee8eda66a29"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-v5vl"
server_key: "157f4950ab1b7ec717c9f6cbbc53d1c6"
server_key_iv: "a6ea1f30880f3209c474aa6980069c6d"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.510891 31430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/wal/instance:
uuid: "5dfaaf818e8447dfb4175ee8eda66a29"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-v5vl"
server_key: "157f4950ab1b7ec717c9f6cbbc53d1c6"
server_key_iv: "a6ea1f30880f3209c474aa6980069c6d"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.515825 31430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 14:55:13.519163 31445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.520423 31430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:55:13.520638 31430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/wal
uuid: "5dfaaf818e8447dfb4175ee8eda66a29"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-v5vl"
server_key: "157f4950ab1b7ec717c9f6cbbc53d1c6"
server_key_iv: "a6ea1f30880f3209c474aa6980069c6d"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.520812 31430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:13.542088 31430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:13.545868 31430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:13.546129 31430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:13.563486 31430 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.253:35483
I20260925 14:55:13.563519 31496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.253:35483 every 8 connection(s)
I20260925 14:55:13.564833 31430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
I20260925 14:55:13.568387 31497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.571857 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 31430
I20260925 14:55:13.572039 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-1/wal/instance
I20260925 14:55:13.572484 17943 external_mini_cluster.cc:1428] Setting key 3f55637a813154ed3de3dce19679fbec
I20260925 14:55:13.574672 31497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:55:13 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.133.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.252@KRBTEST.COM" created.
I20260925 14:55:13.597777 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.577248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.17.133.254:41415)
0925 14:55:13.577471 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.577478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.578746 (+ 1268us) server_negotiation.cc:407] Connection header received
0925 14:55:13.581407 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.581412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.581478 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.581556 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.583613 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.584261 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.586795 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.587201 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.587297 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.591567 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.591571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.591575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.591607 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.594366 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.595150 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.595155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.595159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.595291 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.595789 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.595793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.595795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.596198 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.596371 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.597096 (+ 725us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.597343 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 14:55:13.598397 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.577803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41415 (local address 127.0.0.1:57088)
0925 14:55:13.578593 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.578629 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.581107 (+ 2478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.581817 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.581830 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.582629 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.583437 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.583463 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.584423 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.584432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.586580 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.586596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.587443 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.587453 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.587780 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.588531 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.588571 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.591308 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.594501 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.594511 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.594528 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.595039 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.595400 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.595405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.595408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.595636 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.596342 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.596347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.596729 (+ 382us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.596942 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.597291 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":337,"threads_started":1}
I20260925 14:55:13.600106 31497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:55:13 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
I20260925 14:55:13.614141 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.600940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48354)
0925 14:55:13.601113 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.601130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.601225 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.602112 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.602115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.602503 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.602623 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.602630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.604184 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.604191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.605068 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.605078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.605880 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.605885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.606065 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.606566 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.606582 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.608369 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.611687 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.611692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.611700 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.612071 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.612416 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.612423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.612426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.612570 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.613149 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.613154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.613293 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.613444 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.613733 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 14:55:13.614241 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.601138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.17.133.253:35483)
0925 14:55:13.601748 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.601753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.601775 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:55:13.601811 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.601816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.601970 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.602120 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.602762 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.604036 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.605208 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.605732 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.605866 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.608564 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.608573 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.608632 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.608672 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.611569 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.612193 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.612201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.612205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.612315 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.612689 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.612697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.612700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.613031 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.613248 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.613572 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.613844 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":481,"thread_start_us":361,"threads_started":1}
I20260925 14:55:13.616714 31497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:55:13.617801 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.617288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54578)
0925 14:55:13.617540 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.617665 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 14:55:13.618088 31446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
I20260925 14:55:13.619669 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.619320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54590)
0925 14:55:13.619433 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.619511 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 14:55:13.619890 31497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:37755: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab.
Entry for principal kudu/127.17.133.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab.
I20260925 14:55:13.621937 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.252:37755
--webserver_interface=127.17.133.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab
--principal=kudu/127.17.133.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:41415,127.17.133.253:35483,127.17.133.252:37755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:13.669211 31497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } attempt: 1
I20260925 14:55:13.670377 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.669933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54594)
0925 14:55:13.670137 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.670234 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 14:55:13.671336 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.671016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54602)
0925 14:55:13.671180 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.671240 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 14:55:13.671536 31497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:37755: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
I20260925 14:55:13.739835 31497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } attempt: 2
I20260925 14:55:13.740932 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.740461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54616)
0925 14:55:13.740677 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.740783 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 14:55:13.741840 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.741527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54622)
0925 14:55:13.741678 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.741738 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 14:55:13.742053 31497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:37755: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
I20260925 14:55:13.802883 31423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } attempt: 5
Sep 25 14:55:13 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
I20260925 14:55:13.815871 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.803726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.17.133.253:35483)
0925 14:55:13.803902 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.803910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.803932 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:55:13.804157 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.804162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.804242 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.804348 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.805186 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.805967 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.807097 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.807544 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.807672 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.810732 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.810737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.810740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.810770 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.813397 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.814038 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.814043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.814045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.814161 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.814519 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.814526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.814531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.814805 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.814958 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.815326 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.815543 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:55:13.815871 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.803620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48370)
0925 14:55:13.803849 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.803869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.804009 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.804382 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.804386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.804848 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.805025 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.805033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.806104 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.806111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.806962 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.806973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.807708 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.807714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.807887 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.808543 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.808561 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.810548 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.813534 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.813539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.813543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.813900 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.814272 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.814278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.814281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.814416 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.814922 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.814927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.815092 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.815233 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.815529 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 14:55:13.817190 31423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:55:13.818353 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.817896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54624)
0925 14:55:13.818069 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.818162 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 14:55:13.819322 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.818968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54638)
0925 14:55:13.819144 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.819217 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 14:55:13.819546 31423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:37755: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
I20260925 14:55:13.833230 31497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } attempt: 3
I20260925 14:55:13.834159 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.833770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54646)
0925 14:55:13.833937 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.834020 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 14:55:13.835119 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.834713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54658)
0925 14:55:13.834939 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.835009 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260925 14:55:13.835330 31497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:37755: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
I20260925 14:55:13.870761 31423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } attempt: 1
I20260925 14:55:13.871913 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.871465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54666)
0925 14:55:13.871663 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.871759 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 14:55:13.872989 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.872683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54680)
0925 14:55:13.872811 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.872879 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 14:55:13.873277 31423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:37755: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
W20260925 14:55:13.935423 31504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:13.935685 31504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:13.935868 31504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 14:55:13.941478 31423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } attempt: 2
I20260925 14:55:13.942624 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.942094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54696)
0925 14:55:13.942317 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.942422 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 14:55:13.943420 31504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:13.943538 31504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:13.943615 31504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260925 14:55:13.943640 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.943261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54712)
0925 14:55:13.943406 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.943495 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 14:55:13.943707 31504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:13.943778 31504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:55:13.943850 31423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:37755: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
I20260925 14:55:13.953687 31504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:41415,127.17.133.253:35483,127.17.133.252:37755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab
--principal=kudu/127.17.133.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.252:37755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.133.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:13.955004 31504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:13.956307 31504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:13.967172 31509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.967401 31510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.968062 31512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:13.969000 31504 server_base.cc:1034] running on GCE node
I20260925 14:55:13.969717 31504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:13.971031 31504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:13.972280 31504 hybrid_clock.cc:648] HybridClock initialized: now 1790348113972226 us; error 54 us; skew 500 ppm
I20260925 14:55:13.972718 31504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:13 dist-test-slave-v5vl krb5kdc[31343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.975709 31504 init.cc:377] Logged in from keytab as kudu/127.17.133.252@KRBTEST.COM (short username kudu)
I20260925 14:55:13.977092 31504 webserver.cc:460] Webserver started at http://127.17.133.252:41253/ using document root <none> and password file <none>
I20260925 14:55:13.977744 31504 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:13.977840 31504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:13.978094 31504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:13.978550 31497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } attempt: 4
I20260925 14:55:13.979611 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.979164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54714)
0925 14:55:13.979360 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.979455 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 14:55:13.980540 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.980274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54718)
0925 14:55:13.980390 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.980448 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 14:55:13.980862 31497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:37755: Network error: Client connection negotiation failed: client connection to 127.17.133.252:37755: connect: Connection refused (error 111)
I20260925 14:55:13.981597 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/data/instance:
uuid: "98c48cbf5809405ba6cdae46070859d0"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-v5vl"
server_key: "64ae3eb8b391e1113c8e3e9144317a47"
server_key_iv: "c59155547ebb5f5678b6d38b2166a93e"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.982453 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/wal/instance:
uuid: "98c48cbf5809405ba6cdae46070859d0"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-v5vl"
server_key: "64ae3eb8b391e1113c8e3e9144317a47"
server_key_iv: "c59155547ebb5f5678b6d38b2166a93e"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.987530 31504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:55:13.990770 31519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.992036 31504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 14:55:13.992201 31504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/wal
uuid: "98c48cbf5809405ba6cdae46070859d0"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-v5vl"
server_key: "64ae3eb8b391e1113c8e3e9144317a47"
server_key_iv: "c59155547ebb5f5678b6d38b2166a93e"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.992385 31504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:14.014750 31504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:14.018783 31504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:14.019071 31504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:14.037153 31504 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.252:37755
I20260925 14:55:14.037200 31570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.252:37755 every 8 connection(s)
I20260925 14:55:14.038484 31504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
I20260925 14:55:14.041990 31571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:14.042126 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 31504
I20260925 14:55:14.042331 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347918228911-17943-0/minicluster-data/master-2/wal/instance
I20260925 14:55:14.042758 17943 external_mini_cluster.cc:1428] Setting key 4e84149299bbcb3b16a414bb6e1b506d
I20260925 14:55:14.050055 31423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } attempt: 3
I20260925 14:55:14.051661 31571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:55:14 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
Sep 25 14:55:14 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
Sep 25 14:55:14 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:14.067694 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.050609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54742)
0925 14:55:14.050952 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.050975 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.051130 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.051956 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.051963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.052362 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:14.052504 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.052520 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.054040 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.054045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.055618 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.055632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.056725 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.056731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.056912 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.057508 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:14.057525 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:14.060153 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:14.064986 (+ 4833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.064990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.064993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.065457 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.065885 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.065891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.065894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.066065 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.066689 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:14.066693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:14.066823 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.066977 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.067283 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 14:55:14.068763 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.050626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.17.133.252:37755)
0925 14:55:14.051700 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.051703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.051720 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:55:14.051754 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.051757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.051802 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.051920 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.052699 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.053834 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.055771 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.056531 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.056704 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.061037 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.061041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.061044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.061069 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.064811 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.065577 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.065582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.065585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.065743 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.066188 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.066195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.066198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.066433 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.066595 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.067094 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.067486 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":790,"threads_started":1}
I20260925 14:55:14.068802 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.044556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.17.133.252:37755)
0925 14:55:14.045395 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.045409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.045457 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:55:14.047032 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.047063 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.047542 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.047951 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.049410 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.051248 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.053303 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.054242 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.054406 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.059381 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.059390 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.059401 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.059444 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.063400 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.064214 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.064227 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.064234 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.064421 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.064873 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.064880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.064887 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.065591 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.065921 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.067098 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.067630 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":178,"server-negotiator.queue_time_us":567,"spinlock_wait_cycles":18688,"thread_start_us":346,"threads_started":1}
I20260925 14:55:14.072247 31576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.054382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:41415 (local address 127.0.0.1:57100)
0925 14:55:14.056861 (+ 2479us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.056906 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.057040 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.057736 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.057743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.058479 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:14.058804 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.058820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.060039 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.060048 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.061409 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.061420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.062312 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.062321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.062495 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.063131 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:14.063164 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:14.065943 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:14.069608 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.069617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.069623 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.070198 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.070577 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.070582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.070585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.070733 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.071333 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:14.071340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:14.071492 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.071696 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.071965 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":797,"threads_started":1}
I20260925 14:55:14.073215 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.054424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.17.133.254:41415)
0925 14:55:14.054601 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.054608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.057001 (+ 2393us) server_negotiation.cc:407] Connection header received
0925 14:55:14.057464 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.057469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.057545 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.057659 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.058975 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.059837 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.061548 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.062109 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.062258 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.066117 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.066124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.066129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.066163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.069477 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.070332 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.070337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.070340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.070485 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.070858 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.070865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.070868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.071197 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.071353 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.072744 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.072921 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:55:14.075213 31571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:55:14.077435 31423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb: Bootstrap starting.
I20260925 14:55:14.083096 31423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb: Neither blocks nor log segments found. Creating new log.
Sep 25 14:55:14 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:14.084914 31423 log.cc:824] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:14.092084 31423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb: No bootstrap required, opened a new log
Sep 25 14:55:14 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
I20260925 14:55:14.093539 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.072487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.17.133.254:41415)
0925 14:55:14.073514 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.073518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.073531 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:55:14.073568 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.073571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.073619 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.073729 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.075260 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.076413 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.078891 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.079783 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.079941 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.085851 (+ 5910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.085859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.085866 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.085906 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.090268 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.091096 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.091102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.091106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.091313 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.091725 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.091730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.091734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.092019 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.092326 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.092965 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.093179 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":14714,"threads_started":1}
I20260925 14:55:14.102164 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.076429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.17.133.253:35483)
0925 14:55:14.076633 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.076640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.081984 (+ 5344us) server_negotiation.cc:407] Connection header received
0925 14:55:14.082613 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.082620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.082706 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.082785 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.084675 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.085724 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.089187 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.089788 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.089940 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.093653 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.093662 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.093670 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.093719 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.097778 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.099044 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.099054 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.099060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.099280 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.099827 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.099835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.099841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.100271 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.100437 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.101282 (+ 845us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.101583 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:55:14.104773 31423 raft_consensus.cc:348] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } }
I20260925 14:55:14.105264 31423 raft_consensus.cc:374] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:14.105412 31423 raft_consensus.cc:729] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca71ce1312c46b8abffd4db1b63c6eb, State: Initialized, Role: FOLLOWER
I20260925 14:55:14.106206 31423 consensus_queue.cc:260] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } }
I20260925 14:55:14.106586 31576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.075961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:35483 (local address 127.0.0.1:48380)
0925 14:55:14.081893 (+ 5932us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.081912 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.082299 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.083012 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.083019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.083473 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:14.083650 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.083663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.085950 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.085956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.087379 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.087394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.089961 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.089968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.090211 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.090794 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:14.090815 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:14.093420 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:14.098165 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.098173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.098181 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.098873 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.099456 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.099464 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.099471 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.099682 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.100535 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:14.100542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:14.100922 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.105469 (+ 4547us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.105870 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5819,"thread_start_us":14623,"threads_started":1}
I20260925 14:55:14.107649 31571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:55:14.113507 31581 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.113920 31581 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:14.115731 31423 sys_catalog.cc:564] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 14:55:14 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
Sep 25 14:55:14 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
I20260925 14:55:14.124023 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.095807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.17.133.253:35483)
0925 14:55:14.102826 (+ 7019us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.102834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.102856 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:55:14.102916 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.102925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.103015 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.103146 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.109602 (+ 6456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.111053 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.112406 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.112903 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.113040 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.117822 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.117828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.117833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.117884 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.120652 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.121677 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.121682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.121687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.121845 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.122401 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.122408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.122413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.122721 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.123034 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.123370 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.123611 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6864,"thread_start_us":15250,"threads_started":1}
I20260925 14:55:14.128428 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.108636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54754)
0925 14:55:14.109139 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.109161 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.109327 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.109848 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.109855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.110587 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:14.110953 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.110967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.113917 (+ 2950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.113923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.115346 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.115362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.116477 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.116483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.116718 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.118596 (+ 1878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:14.118628 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:14.121455 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:14.125683 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.125687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.125689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.126031 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.126421 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.126429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.126435 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.126615 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.127440 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:14.127446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:14.127626 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.127780 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.128080 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341}
I20260925 14:55:14.129300 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.108863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.17.133.252:37755)
0925 14:55:14.109079 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.109088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.109235 (+ 147us) server_negotiation.cc:407] Connection header received
0925 14:55:14.109445 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.109453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.109543 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.109664 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.111167 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.112521 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.115501 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.116175 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.116307 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.121645 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.121653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.121657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.121690 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.125565 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.126162 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.126167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.126171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.126294 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.126751 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.126759 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.126765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.127229 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.127392 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.128762 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.129003 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:55:14.131044 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 14:55:14.134377 31592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:55:14.134518 31592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:55:14.135010 31571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0: Bootstrap starting.
I20260925 14:55:14.138799 31571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:14.140025 31571 log.cc:824] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:14.135950 31356 master_runner.cc:186] Error getting master registration for 127.17.133.253:35483: OK, instance_id { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" instance_seqno: 1790348113558821 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:55:14.149587 31356 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:55:14.152415 31571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0: No bootstrap required, opened a new log
I20260925 14:55:14.166090 31571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } }
I20260925 14:55:14.166599 31571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:14.166726 31571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c48cbf5809405ba6cdae46070859d0, State: Initialized, Role: FOLLOWER
I20260925 14:55:14.167605 31571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } }
I20260925 14:55:14.174150 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.155078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.17.133.253:35483)
0925 14:55:14.155287 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.155293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.155700 (+ 407us) server_negotiation.cc:407] Connection header received
0925 14:55:14.155990 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.155995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.156078 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.156155 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.158594 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.159756 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.162689 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.163353 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.163456 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.167252 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.167260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.167265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.167309 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.170846 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.171747 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.171752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.171756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.171917 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.172351 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.172356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.172359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.172746 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.172970 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.173549 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.173777 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:55:14.180914 31602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.181392 31602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:14.183250 31571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:14.191805 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.155903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.17.133.252:37755)
0925 14:55:14.156078 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.156084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.175215 (+ 19131us) server_negotiation.cc:407] Connection header received
0925 14:55:14.175445 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.175451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.175541 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.175608 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.177411 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.178550 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.180663 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.181363 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.181454 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.184942 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.184948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.184970 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.185017 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.188386 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.189348 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.189355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.189359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.189493 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.189988 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.189994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.189997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.190375 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.190638 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.191188 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.191431 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:55:14.198367 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.161923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.17.133.254:41415)
0925 14:55:14.162104 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.162127 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.177815 (+ 15688us) server_negotiation.cc:407] Connection header received
0925 14:55:14.178288 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.178296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.178379 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.178454 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.181177 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.182214 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.184573 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.185478 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.185609 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.189963 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.189977 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.189985 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.190038 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.194354 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.195707 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.195712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.195716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.195882 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.196477 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.196485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.196489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.196941 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.197220 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.197697 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.197961 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 14:55:14.208973 31613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:55:14.209098 31613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:55:14.207989 31504 master_runner.cc:186] Error getting master registration for 127.17.133.253:35483: OK, instance_id { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" instance_seqno: 1790348113558821 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:55:14.220562 31504 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 14:55:14.271073 31525 tablet.cc:2369] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:14.280081 31497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } attempt: 5
Sep 25 14:55:14 dist-test-slave-v5vl krb5kdc[31343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
I20260925 14:55:14.298362 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.280683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:37755 (local address 127.0.0.1:54760)
0925 14:55:14.281007 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.281028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.281130 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.281971 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.281976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.282497 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:14.282709 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.282721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.284190 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.284196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.285682 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.285697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.286638 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.286645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.286847 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.287462 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:14.287502 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:14.290261 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:14.295502 (+ 5241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.295509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.295514 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.296013 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.296460 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.296465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.296468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.296694 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.297336 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:14.297342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:14.297537 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.297691 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.297980 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260925 14:55:14.298395 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.281406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.17.133.252:37755)
0925 14:55:14.281605 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.281611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.281628 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:55:14.281680 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.281685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.281797 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.281918 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.282861 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.284022 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.285831 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.286486 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.286614 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.290438 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.290445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.290450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.290492 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.295354 (+ 4862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.296146 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.296152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.296156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.296341 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.296800 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.296806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.296809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.297209 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.297345 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.297816 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.298063 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:55:14.304329 31497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29: Bootstrap starting.
I20260925 14:55:14.309659 31497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:14.310879 31497 log.cc:824] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:14.314042 31497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29: No bootstrap required, opened a new log
I20260925 14:55:14.326915 31497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } }
I20260925 14:55:14.327337 31497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:14.327454 31497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dfaaf818e8447dfb4175ee8eda66a29, State: Initialized, Role: FOLLOWER
I20260925 14:55:14.328122 31497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } }
W20260925 14:55:14.328920 31377 tablet.cc:2369] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:14.330057 31616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.330502 31616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:14.331308 31497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:14.338337 31602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:14.338653 31602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } }
I20260925 14:55:14.340179 31602 leader_election.cc:288] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eca71ce1312c46b8abffd4db1b63c6eb (127.17.133.254:41415), 5dfaaf818e8447dfb4175ee8eda66a29 (127.17.133.253:35483)
I20260925 14:55:14.340890 31398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98c48cbf5809405ba6cdae46070859d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" is_pre_election: true
I20260925 14:55:14.341054 31472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98c48cbf5809405ba6cdae46070859d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" is_pre_election: true
I20260925 14:55:14.341461 31398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c48cbf5809405ba6cdae46070859d0 in term 0.
I20260925 14:55:14.341532 31472 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c48cbf5809405ba6cdae46070859d0 in term 0.
I20260925 14:55:14.342163 31522 leader_election.cc:302] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dfaaf818e8447dfb4175ee8eda66a29, 98c48cbf5809405ba6cdae46070859d0; no voters:
I20260925 14:55:14.343611 31602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:14.343757 31602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:14.343869 31602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:14.347117 31602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } }
I20260925 14:55:14.348093 31398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98c48cbf5809405ba6cdae46070859d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca71ce1312c46b8abffd4db1b63c6eb"
I20260925 14:55:14.348294 31398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:14.348307 31472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98c48cbf5809405ba6cdae46070859d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dfaaf818e8447dfb4175ee8eda66a29"
I20260925 14:55:14.348485 31472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:14.351579 31472 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c48cbf5809405ba6cdae46070859d0 in term 1.
I20260925 14:55:14.351665 31398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c48cbf5809405ba6cdae46070859d0 in term 1.
I20260925 14:55:14.351949 31522 leader_election.cc:302] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dfaaf818e8447dfb4175ee8eda66a29, 98c48cbf5809405ba6cdae46070859d0; no voters:
I20260925 14:55:14.354660 31602 leader_election.cc:288] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [CANDIDATE]: Term 1 election: Requested vote from peers eca71ce1312c46b8abffd4db1b63c6eb (127.17.133.254:41415), 5dfaaf818e8447dfb4175ee8eda66a29 (127.17.133.253:35483)
I20260925 14:55:14.354849 31602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:14.354976 31602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [term 1 LEADER]: Becoming Leader. State: Replica: 98c48cbf5809405ba6cdae46070859d0, State: Running, Role: LEADER
I20260925 14:55:14.355479 31602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } }
I20260925 14:55:14.372836 31622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c48cbf5809405ba6cdae46070859d0. Latest consensus state: current_term: 1 leader_uuid: "98c48cbf5809405ba6cdae46070859d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.373250 31622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:14.377272 31626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:14.379652 31626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:14.399273 31472 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [term 1 FOLLOWER]: Refusing update from remote peer 98c48cbf5809405ba6cdae46070859d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:14.400182 31622 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:14.406847 31398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [term 1 FOLLOWER]: Refusing update from remote peer 98c48cbf5809405ba6cdae46070859d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:14.407907 31622 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:14.432387 31629 mvcc.cc:205] Tried to move back new op lower bound from 7333265876567199744 to 7333265876412833792. Current Snapshot: MvccSnapshot[applied={T|T < 7333265876567199744}]
I20260925 14:55:14.435801 31616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c48cbf5809405ba6cdae46070859d0. Latest consensus state: current_term: 1 leader_uuid: "98c48cbf5809405ba6cdae46070859d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.436174 31616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:14.439510 31602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98c48cbf5809405ba6cdae46070859d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.439807 31602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:14.442101 31626 catalog_manager.cc:1347] Generated new cluster ID: 7006179aab474e49bd1a15b6da2d9b2c
I20260925 14:55:14.442196 31626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:14.443130 31581 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c48cbf5809405ba6cdae46070859d0. Latest consensus state: current_term: 1 leader_uuid: "98c48cbf5809405ba6cdae46070859d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.443636 31581 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:14.448004 31622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98c48cbf5809405ba6cdae46070859d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.448273 31622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:14.450767 31637 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:55:14.457109 31581 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98c48cbf5809405ba6cdae46070859d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.457386 31581 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:14.459545 31616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98c48cbf5809405ba6cdae46070859d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca71ce1312c46b8abffd4db1b63c6eb" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 41415 } } peers { permanent_uuid: "5dfaaf818e8447dfb4175ee8eda66a29" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 35483 } } peers { permanent_uuid: "98c48cbf5809405ba6cdae46070859d0" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 37755 } } }
I20260925 14:55:14.459824 31616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:14.465565 31626 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:14.467444 31626 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:14.493026 31626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98c48cbf5809405ba6cdae46070859d0: Generated new TSK 0
I20260925 14:55:14.494019 31626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:55:15.150542 31592 catalog_manager.cc:1259] Loaded cluster ID: 7006179aab474e49bd1a15b6da2d9b2c
I20260925 14:55:15.150647 31592 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb: loading cluster ID for follower catalog manager: success
I20260925 14:55:15.153206 31592 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb: acquiring CA information for follower catalog manager: success
I20260925 14:55:15.154937 31592 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eca71ce1312c46b8abffd4db1b63c6eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:55:15.455101 31637 catalog_manager.cc:1259] Loaded cluster ID: 7006179aab474e49bd1a15b6da2d9b2c
I20260925 14:55:15.455252 31637 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29: loading cluster ID for follower catalog manager: success
I20260925 14:55:15.457700 31637 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29: acquiring CA information for follower catalog manager: success
I20260925 14:55:15.459460 31637 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5dfaaf818e8447dfb4175ee8eda66a29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:55:15.578588 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 31356
I20260925 14:55:15.580132 31418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:15.690584 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 31356
I20260925 14:55:15.711035 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 31430
I20260925 14:55:15.712142 31492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:15.822976 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 31430
I20260925 14:55:15.841617 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 31504
I20260925 14:55:15.842576 31566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:15.948153 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 31504
2026-09-25T14:55:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4552 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:16 dist-test-slave-v5vl krb5kdc[31650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:16 dist-test-slave-v5vl krb5kdc[31650](info): set up 2 sockets
Sep 25 14:55:16 dist-test-slave-v5vl krb5kdc[31650](info): commencing operation
krb5kdc: starting...
W20260925 14:55:17.037278 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.009s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:55:17 dist-test-slave-v5vl krb5kdc[31650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:55:17.246099 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:44095
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:44095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:17.557564 31663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:17.557934 31663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:17.558125 31663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:17.565251 31663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:17.565384 31663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:17.565490 31663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:17.565598 31663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:17.565685 31663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:17.574903 31663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:44095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:44095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:17.576157 31663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:17.577492 31663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:17.588469 31668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:17.588415 31669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:17.588685 31671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:17.589502 31663 server_base.cc:1034] running on GCE node
I20260925 14:55:17.590217 31663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:17.591408 31663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:17.592676 31663 hybrid_clock.cc:648] HybridClock initialized: now 1790348117592619 us; error 47 us; skew 500 ppm
I20260925 14:55:17.593006 31663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:17 dist-test-slave-v5vl krb5kdc[31650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348117, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:17.601019 31663 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:55:17.602363 31663 webserver.cc:460] Webserver started at http://127.17.133.254:33521/ using document root <none> and password file <none>
I20260925 14:55:17.602969 31663 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:17.603061 31663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:17.603293 31663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:17.606068 31663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "aecc8425b7424559aab88ac13e3a2344"
format_stamp: "Formatted at 2026-09-25 14:55:17 on dist-test-slave-v5vl"
server_key: "565deda7a4f6b3f34e4bee691c5123f2"
server_key_iv: "1f3077aa5a425173bf3a23244a86e147"
server_key_version: "encryptionkey@0"
I20260925 14:55:17.606714 31663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "aecc8425b7424559aab88ac13e3a2344"
format_stamp: "Formatted at 2026-09-25 14:55:17 on dist-test-slave-v5vl"
server_key: "565deda7a4f6b3f34e4bee691c5123f2"
server_key_iv: "1f3077aa5a425173bf3a23244a86e147"
server_key_version: "encryptionkey@0"
I20260925 14:55:17.611572 31663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:55:17.615015 31678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:17.616282 31663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:55:17.616465 31663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "aecc8425b7424559aab88ac13e3a2344"
format_stamp: "Formatted at 2026-09-25 14:55:17 on dist-test-slave-v5vl"
server_key: "565deda7a4f6b3f34e4bee691c5123f2"
server_key_iv: "1f3077aa5a425173bf3a23244a86e147"
server_key_version: "encryptionkey@0"
I20260925 14:55:17.616743 31663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:17.634477 31663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:17.638242 31663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:17.638523 31663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:17.656213 31663 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:44095
I20260925 14:55:17.656234 31729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:44095 every 8 connection(s)
I20260925 14:55:17.657526 31663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:17.661029 31730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:17.665160 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 31663
I20260925 14:55:17.665381 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:55:17.666182 17943 external_mini_cluster.cc:1428] Setting key 7c77c78d8edc99d96461c443367b09d8
I20260925 14:55:17.672008 31730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344: Bootstrap starting.
I20260925 14:55:17.675482 31730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:17.676681 31730 log.cc:824] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:17.679281 31730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344: No bootstrap required, opened a new log
Sep 25 14:55:17 dist-test-slave-v5vl krb5kdc[31650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:17.688151 31730 raft_consensus.cc:348] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecc8425b7424559aab88ac13e3a2344" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44095 } }
I20260925 14:55:17.688513 31730 raft_consensus.cc:374] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:17.688657 31730 raft_consensus.cc:729] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecc8425b7424559aab88ac13e3a2344, State: Initialized, Role: FOLLOWER
I20260925 14:55:17.688767 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:17.667875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.17.133.254:44095)
0925 14:55:17.668701 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:17.668715 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:17.668762 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:55:17.670692 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:17.670712 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:17.671059 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:17.671343 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:17.673044 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:17.674334 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:17.676226 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:17.676997 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:17.677186 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:17.681555 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:17.681565 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:17.681577 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:17.681613 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:17.684739 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:17.685501 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:17.685507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:17.685512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:17.685667 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:17.686059 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:17.686062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:17.686063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:17.686578 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:17.686777 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:17.687224 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 14:55:17.687594 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":370,"threads_started":1}
I20260925 14:55:17.689288 31730 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecc8425b7424559aab88ac13e3a2344" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44095 } }
I20260925 14:55:17.689580 31730 raft_consensus.cc:388] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:17.689754 31730 raft_consensus.cc:482] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:17.689940 31730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:17.693007 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:55:17.693154 31730 raft_consensus.cc:504] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecc8425b7424559aab88ac13e3a2344" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44095 } }
I20260925 14:55:17.693657 31730 leader_election.cc:302] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aecc8425b7424559aab88ac13e3a2344; no voters:
I20260925 14:55:17.694263 31730 leader_election.cc:288] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:17.694410 31735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:17.695001 31735 raft_consensus.cc:686] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [term 1 LEADER]: Becoming Leader. State: Replica: aecc8425b7424559aab88ac13e3a2344, State: Running, Role: LEADER
I20260925 14:55:17.695654 31735 consensus_queue.cc:237] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecc8425b7424559aab88ac13e3a2344" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44095 } }
I20260925 14:55:17.697734 31730 sys_catalog.cc:564] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:17.698807 31736 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aecc8425b7424559aab88ac13e3a2344. Latest consensus state: current_term: 1 leader_uuid: "aecc8425b7424559aab88ac13e3a2344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecc8425b7424559aab88ac13e3a2344" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44095 } } }
I20260925 14:55:17.699330 31736 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:17.698796 31737 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aecc8425b7424559aab88ac13e3a2344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecc8425b7424559aab88ac13e3a2344" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44095 } } }
I20260925 14:55:17.705057 31737 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344 [sys.catalog]: This master's current role is: LEADER
W20260925 14:55:17.708101 31751 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:55:17.708217 31751 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:55:17.708374 31741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:17.710531 31741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:17.719277 31741 catalog_manager.cc:1347] Generated new cluster ID: eb11e2c5c5a34bd08efd4b2070fdfbda
I20260925 14:55:17.719395 31741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:17.732797 31741 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:17.734064 31741 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:17.751163 31741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aecc8425b7424559aab88ac13e3a2344: Generated new TSK 0
I20260925 14:55:17.752031 31741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:55:17.805985 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 31663
I20260925 14:55:17.806977 31725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:17.902348 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 31663
2026-09-25T14:55:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1949 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:17 dist-test-slave-v5vl krb5kdc[31758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:17 dist-test-slave-v5vl krb5kdc[31758](info): set up 2 sockets
Sep 25 14:55:17 dist-test-slave-v5vl krb5kdc[31758](info): commencing operation
krb5kdc: starting...
W20260925 14:55:18.961254 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:55:19 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:55:19.184000 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34843
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:19.498181 31771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:19.498507 31771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:19.498661 31771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:19.505790 31771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:19.505928 31771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:19.506034 31771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:19.506141 31771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:19.506206 31771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:19.515780 31771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:19.517057 31771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:19.518435 31771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:19.529479 31777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:19.529604 31776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:19.529803 31779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:19.530447 31771 server_base.cc:1034] running on GCE node
I20260925 14:55:19.531108 31771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:19.532325 31771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:19.533537 31771 hybrid_clock.cc:648] HybridClock initialized: now 1790348119533489 us; error 35 us; skew 500 ppm
I20260925 14:55:19.533842 31771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:19 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348119, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:19.536681 31771 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:55:19.538036 31771 webserver.cc:460] Webserver started at http://127.17.133.254:35993/ using document root <none> and password file <none>
I20260925 14:55:19.538719 31771 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:19.538861 31771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:19.539143 31771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:19.541836 31771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "c9f316ce510d4ea0a10a92384d4b0d80"
format_stamp: "Formatted at 2026-09-25 14:55:19 on dist-test-slave-v5vl"
server_key: "93530d2e1fb257f80196efc1039f07a4"
server_key_iv: "da66193076bba194290c3d53bc4c7835"
server_key_version: "encryptionkey@0"
I20260925 14:55:19.542547 31771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "c9f316ce510d4ea0a10a92384d4b0d80"
format_stamp: "Formatted at 2026-09-25 14:55:19 on dist-test-slave-v5vl"
server_key: "93530d2e1fb257f80196efc1039f07a4"
server_key_iv: "da66193076bba194290c3d53bc4c7835"
server_key_version: "encryptionkey@0"
I20260925 14:55:19.547099 31771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:55:19.550185 31786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:19.551363 31771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:19.551554 31771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "c9f316ce510d4ea0a10a92384d4b0d80"
format_stamp: "Formatted at 2026-09-25 14:55:19 on dist-test-slave-v5vl"
server_key: "93530d2e1fb257f80196efc1039f07a4"
server_key_iv: "da66193076bba194290c3d53bc4c7835"
server_key_version: "encryptionkey@0"
I20260925 14:55:19.551738 31771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:19.561831 31771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:19.565562 31771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:19.565845 31771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:19.583457 31771 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:34843
I20260925 14:55:19.583492 31837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:34843 every 8 connection(s)
I20260925 14:55:19.584787 31771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:19.588279 31838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:19.593225 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 31771
I20260925 14:55:19.593443 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:55:19.594220 17943 external_mini_cluster.cc:1428] Setting key b979270435987dd22bbcc5eb29b52d8e
I20260925 14:55:19.599532 31838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Bootstrap starting.
I20260925 14:55:19.603094 31838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:19.604295 31838 log.cc:824] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:19.607628 31838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: No bootstrap required, opened a new log
Sep 25 14:55:19 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:19.615840 31838 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:19.616284 31838 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:19.616416 31838 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f316ce510d4ea0a10a92384d4b0d80, State: Initialized, Role: FOLLOWER
I20260925 14:55:19.617034 31838 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:19.617146 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:19.596261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.17.133.254:34843)
0925 14:55:19.597105 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:19.597116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:19.597161 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:55:19.598794 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:19.598815 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:19.599168 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:19.599470 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:19.601217 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:19.602460 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:19.604349 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:19.605462 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:19.605641 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:19.610059 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:19.610069 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:19.610081 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:19.610115 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:19.613142 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:19.613799 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:19.613810 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:19.613815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:19.613980 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:19.614417 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:19.614424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:19.614431 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:19.614965 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:19.615297 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:19.615683 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 14:55:19.616074 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":428,"threads_started":1}
I20260925 14:55:19.617309 31838 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:19.617491 31838 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:19.617671 31838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:19.620332 31838 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:19.620772 31838 leader_election.cc:302] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9f316ce510d4ea0a10a92384d4b0d80; no voters:
I20260925 14:55:19.621222 31838 leader_election.cc:288] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:19.621309 31843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:19.621999 31843 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 1 LEADER]: Becoming Leader. State: Replica: c9f316ce510d4ea0a10a92384d4b0d80, State: Running, Role: LEADER
I20260925 14:55:19.622522 31843 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:19.623142 31838 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:19.625345 31845 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9f316ce510d4ea0a10a92384d4b0d80. Latest consensus state: current_term: 1 leader_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } } }
I20260925 14:55:19.625773 31845 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:19.625638 31844 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } } }
I20260925 14:55:19.625941 31844 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:19.627192 31848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:19.632562 31848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:19.642833 31848 catalog_manager.cc:1347] Generated new cluster ID: ceab647ffa544bb08e4ed9c8d3b8c539
I20260925 14:55:19.642943 31848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:19.669746 31848 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:19.671520 31848 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:19.692070 31848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Generated new TSK 0
I20260925 14:55:19.693261 31848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:55:19.760270 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34843
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:20.078003 31864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:20.078303 31864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:20.078506 31864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:20.085374 31864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:20.085470 31864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:20.085664 31864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:55:20.094893 31864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:20.096112 31864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:20.097435 31864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:20.109933 31869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:20.109905 31872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:20.109967 31870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:20.110747 31864 server_base.cc:1034] running on GCE node
I20260925 14:55:20.111239 31864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:20.112047 31864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:20.113265 31864 hybrid_clock.cc:648] HybridClock initialized: now 1790348120113219 us; error 52 us; skew 500 ppm
I20260925 14:55:20.113557 31864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:20 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:20.116524 31864 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:55:20.117955 31864 webserver.cc:460] Webserver started at http://127.17.133.193:33279/ using document root <none> and password file <none>
I20260925 14:55:20.118640 31864 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:20.118785 31864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:20.119073 31864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:20.121891 31864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "2cee73d048c4465a94507b827d256797"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "ba1f4bcd96ae8ea60c2ac2493ba6ef73"
server_key_iv: "9b28511edb31e21a536412b265efe175"
server_key_version: "encryptionkey@0"
I20260925 14:55:20.122637 31864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "2cee73d048c4465a94507b827d256797"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "ba1f4bcd96ae8ea60c2ac2493ba6ef73"
server_key_iv: "9b28511edb31e21a536412b265efe175"
server_key_version: "encryptionkey@0"
I20260925 14:55:20.127789 31864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 14:55:20.131104 31879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:20.132365 31864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:55:20.132485 31864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "2cee73d048c4465a94507b827d256797"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "ba1f4bcd96ae8ea60c2ac2493ba6ef73"
server_key_iv: "9b28511edb31e21a536412b265efe175"
server_key_version: "encryptionkey@0"
I20260925 14:55:20.132712 31864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:20.167039 31864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:20.172289 31864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:20.172735 31864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:20.174072 31864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:20.176046 31864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:20.176131 31864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:20.176286 31864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:20.176350 31864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:20.208244 31864 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:41405
I20260925 14:55:20.208353 31991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:41405 every 8 connection(s)
I20260925 14:55:20.209565 31864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:20.213428 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 31864
I20260925 14:55:20.213624 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:20.214360 17943 external_mini_cluster.cc:1428] Setting key 903561e7bc84a48c2600e863118cc559
Sep 25 14:55:20 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:20.229987 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:20.211976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:45815 (local address 127.17.133.254:34843)
0925 14:55:20.212444 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:20.212451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:20.213470 (+ 1019us) server_negotiation.cc:407] Connection header received
0925 14:55:20.215531 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:20.215537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:20.215610 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:20.215724 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:20.217313 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.218024 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.219612 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.220408 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.220554 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:20.224132 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:20.224136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:20.224140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:20.224170 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:20.226756 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:20.227475 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:20.227480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:20.227487 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:20.227599 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:20.228016 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:20.228021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:20.228024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:20.228290 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:20.228466 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:20.229217 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 14:55:20.229475 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20260925 14:55:20.230654 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:20.212558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:45815)
0925 14:55:20.213311 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:20.213354 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:20.215191 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:20.215923 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:20.215931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:20.216520 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:20.217149 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:20.217168 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.218190 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.218196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:20.219433 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:20.219451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.220666 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.220672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:20.220870 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:20.221544 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:20.221569 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:20.223933 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:20.226938 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:20.226946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:20.226962 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:20.227353 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:20.227702 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:20.227707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:20.227710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:20.227903 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:20.228415 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:20.228422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:20.228816 (+ 394us) client_negotiation.cc:769] Sending connection context
0925 14:55:20.229066 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 14:55:20.229506 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":301,"threads_started":1}
I20260925 14:55:20.232136 31992 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34843
I20260925 14:55:20.232458 31992 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:20.233286 31992 heartbeater.cc:511] Master 127.17.133.254:34843 requested a full tablet report, sending...
I20260925 14:55:20.235344 31803 ts_manager.cc:194] Registered new tserver with Master: 2cee73d048c4465a94507b827d256797 (127.17.133.193:41405)
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:55:20.236950 31803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:45815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:55:20.263207 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34843
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:20.563225 31998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:20.563490 31998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:20.563580 31998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:20.570587 31998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:20.570683 31998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:20.570796 31998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:55:20.579571 31998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:20.580681 31998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:20.581876 31998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:20.594728 32006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:20.594704 32004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:20.595048 32003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:20.595491 31998 server_base.cc:1034] running on GCE node
I20260925 14:55:20.596112 31998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:20.597159 31998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:20.598378 31998 hybrid_clock.cc:648] HybridClock initialized: now 1790348120598323 us; error 47 us; skew 500 ppm
I20260925 14:55:20.598701 31998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:20 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:20.602072 31998 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:55:20.603379 31998 webserver.cc:460] Webserver started at http://127.17.133.194:38207/ using document root <none> and password file <none>
I20260925 14:55:20.604044 31998 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:20.604171 31998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:20.604473 31998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:20.607317 31998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "8bf0a857b7a24b13bc3fb4f4011a008a"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "016aece6c0dd93f9b8208389bbe39cc5"
server_key_iv: "bbac9f056e2738662d23dc9591f1df20"
server_key_version: "encryptionkey@0"
I20260925 14:55:20.608070 31998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "8bf0a857b7a24b13bc3fb4f4011a008a"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "016aece6c0dd93f9b8208389bbe39cc5"
server_key_iv: "bbac9f056e2738662d23dc9591f1df20"
server_key_version: "encryptionkey@0"
I20260925 14:55:20.612723 31998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 14:55:20.615775 32013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:20.617233 31998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:55:20.617399 31998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "8bf0a857b7a24b13bc3fb4f4011a008a"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "016aece6c0dd93f9b8208389bbe39cc5"
server_key_iv: "bbac9f056e2738662d23dc9591f1df20"
server_key_version: "encryptionkey@0"
I20260925 14:55:20.617571 31998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:20.642715 31998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:20.646359 31998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:20.646623 31998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:20.647475 31998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:20.648731 31998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:20.648802 31998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:20.648967 31998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:20.649019 31998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:20.680540 31998 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:34233
I20260925 14:55:20.680560 32125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:34233 every 8 connection(s)
I20260925 14:55:20.681842 31998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:20.690801 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 31998
I20260925 14:55:20.690991 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:20.691433 17943 external_mini_cluster.cc:1428] Setting key 2b40c6cceaf7b9d3920aa9a391c9b6ef
Sep 25 14:55:20 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348120, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:20.700744 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:20.684274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:53657 (local address 127.17.133.254:34843)
0925 14:55:20.684516 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:20.684523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:20.685682 (+ 1159us) server_negotiation.cc:407] Connection header received
0925 14:55:20.687297 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:20.687301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:20.687366 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:20.687452 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:20.688813 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.689537 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.690945 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.691396 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.691527 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:20.695348 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:20.695353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:20.695356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:20.695386 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:20.697839 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:20.698551 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:20.698563 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:20.698565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:20.698697 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:20.699115 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:20.699124 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:20.699126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:20.699374 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:20.699538 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:20.700167 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 14:55:20.700392 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 14:55:20.701964 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:20.684856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:53657)
0925 14:55:20.685529 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:20.685570 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:20.687060 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:20.687645 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:20.687652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:20.688104 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:20.688674 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:20.688691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.689661 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.689664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:20.690799 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:20.690810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.691605 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.691612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:20.691828 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:20.693134 (+ 1306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:20.693154 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:20.695148 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:20.697970 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:20.697978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:20.697992 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:20.698425 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:20.698814 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:20.698820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:20.698823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:20.698998 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:20.699515 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:20.699522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:20.699829 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 14:55:20.700070 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 14:55:20.700545 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":265,"threads_started":1}
I20260925 14:55:20.703323 32126 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34843
I20260925 14:55:20.703636 32126 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:20.704447 32126 heartbeater.cc:511] Master 127.17.133.254:34843 requested a full tablet report, sending...
I20260925 14:55:20.705865 31803 ts_manager.cc:194] Registered new tserver with Master: 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233)
I20260925 14:55:20.706640 31803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:53657
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:55:20.740917 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34843
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:21.049628 32131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:21.049961 32131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:21.050145 32131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:21.057359 32131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:21.057452 32131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:21.057649 32131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:55:21.066962 32131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:21.068238 32131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:21.069581 32131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:21.082001 32136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:21.082044 32137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:21.082074 32139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:21.082921 32131 server_base.cc:1034] running on GCE node
I20260925 14:55:21.083408 32131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:21.084245 32131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:21.085426 32131 hybrid_clock.cc:648] HybridClock initialized: now 1790348121085371 us; error 48 us; skew 500 ppm
I20260925 14:55:21.085670 32131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:21 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348121, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:21.091493 32131 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:55:21.092800 32131 webserver.cc:460] Webserver started at http://127.17.133.195:35661/ using document root <none> and password file <none>
I20260925 14:55:21.093405 32131 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:21.093497 32131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:21.093720 32131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:21.096774 32131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "bb67611044ac45ab8826731ed1522784"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-v5vl"
server_key: "a4a5157e042de1a84367eac05db83165"
server_key_iv: "3cfad6ec043f2a189d9b3908f9e6d1b6"
server_key_version: "encryptionkey@0"
I20260925 14:55:21.097908 32131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "bb67611044ac45ab8826731ed1522784"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-v5vl"
server_key: "a4a5157e042de1a84367eac05db83165"
server_key_iv: "3cfad6ec043f2a189d9b3908f9e6d1b6"
server_key_version: "encryptionkey@0"
I20260925 14:55:21.105741 32131 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260925 14:55:21.111454 32146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:21.113116 32131 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 14:55:21.113274 32131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "bb67611044ac45ab8826731ed1522784"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-v5vl"
server_key: "a4a5157e042de1a84367eac05db83165"
server_key_iv: "3cfad6ec043f2a189d9b3908f9e6d1b6"
server_key_version: "encryptionkey@0"
I20260925 14:55:21.113523 32131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:21.136770 32131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:21.141978 32131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:21.142251 32131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:21.143319 32131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:21.144910 32131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:21.145005 32131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:21.145112 32131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:21.145183 32131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:21.177309 32131 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:46749
I20260925 14:55:21.177356 32258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:46749 every 8 connection(s)
I20260925 14:55:21.178558 32131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:21.180665 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 32131
I20260925 14:55:21.180842 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:21.181394 17943 external_mini_cluster.cc:1428] Setting key 8e8f3f542e07cb82694dc0ea77921b4f
Sep 25 14:55:21 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348121, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:21.196530 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.181019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:51733 (local address 127.17.133.254:34843)
0925 14:55:21.181239 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.181245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.182567 (+ 1322us) server_negotiation.cc:407] Connection header received
0925 14:55:21.184217 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.184222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.184292 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.184365 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:21.185660 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.186387 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.187830 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.188205 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.188291 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.191418 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:21.191422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:21.191425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:21.191455 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:21.193867 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.194465 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.194476 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.194479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.194621 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.195009 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.195014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.195019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.195261 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:21.195414 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.196001 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.196190 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 14:55:21.197479 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.181609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:51733)
0925 14:55:21.182415 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:21.182468 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:21.183983 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:21.184537 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:21.184543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:21.185033 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:21.185534 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.185549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.186508 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.186512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.187709 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.187719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.188401 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.188406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.188644 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.189201 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:21.189225 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:21.191245 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:21.193987 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:21.193992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:21.194001 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:21.194354 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:21.194727 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:21.194730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:21.194732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:21.194895 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:21.195385 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:21.195391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:21.195662 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 14:55:21.195905 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 14:55:21.196328 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":351,"threads_started":1}
I20260925 14:55:21.198630 32259 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34843
I20260925 14:55:21.198910 32259 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:21.199604 32259 heartbeater.cc:511] Master 127.17.133.254:34843 requested a full tablet report, sending...
I20260925 14:55:21.200923 31803 ts_manager.cc:194] Registered new tserver with Master: bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749)
I20260925 14:55:21.201661 31803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:51733
I20260925 14:55:21.207876 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:21.225224 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.213183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.17.133.254:34843)
0925 14:55:21.213377 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.213383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.213821 (+ 438us) server_negotiation.cc:407] Connection header received
0925 14:55:21.213975 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.213979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.214057 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.214126 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:21.215451 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.216108 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.217415 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.217837 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.217927 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.220160 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:21.220164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:21.220167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:21.220194 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:21.223008 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.223564 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.223568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.223570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.223678 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.223969 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.223972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.223974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.224244 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:21.224427 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.224758 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.224905 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:55:21.228822 31803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:21.235467 31803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:21.239838 31992 heartbeater.cc:503] Master 127.17.133.254:34843 was elected leader, sending a full tablet report...
I20260925 14:55:21.260660 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.247301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:46749 (local address 127.0.0.1:56210)
0925 14:55:21.249057 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:21.249073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:21.249186 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:21.252388 (+ 3202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:21.252399 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:21.252439 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:21.253006 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.253021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.255594 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.255603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.258246 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.258263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.259530 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.259536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.259745 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.259823 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 14:55:21.259959 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 14:55:21.260147 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"mutex_wait_us":176,"thread_start_us":308,"threads_started":1}
I20260925 14:55:21.260910 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.247751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.17.133.195:46749)
0925 14:55:21.251665 (+ 3914us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.251674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.251714 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:55:21.251807 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.251814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.252118 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.253231 (+ 1113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:21.253551 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.255326 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.258443 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.259323 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.259528 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.259718 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.260359 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.260514 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3751,"thread_start_us":3586,"threads_started":1}
I20260925 14:55:21.264885 32194 tablet_service.cc:1467] Processing CreateTablet for tablet e8d430fed3c846fbb30b3ee6c67cee3f (DEFAULT_TABLE table=test-table [id=c0af5a5ad6f64d899d26138ee2460df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:21.266376 32194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d430fed3c846fbb30b3ee6c67cee3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:21.267938 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.249004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.17.133.194:34233)
0925 14:55:21.257339 (+ 8335us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.257350 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.260145 (+ 2795us) server_negotiation.cc:407] Connection header received
0925 14:55:21.260390 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.260398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.260660 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.260862 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:21.261938 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.263154 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.265477 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.266680 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.266859 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.267024 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.267496 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.267642 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8131,"thread_start_us":7975,"threads_started":1}
I20260925 14:55:21.267998 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.248517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:34233 (local address 127.0.0.1:47822)
0925 14:55:21.260012 (+ 11495us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:21.260029 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:21.260192 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:21.261103 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:21.261107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:21.261139 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:21.261725 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.261739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.263291 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.263296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.265314 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.265328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.267162 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.267169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.267327 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.267350 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:55:21.267455 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:55:21.267571 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11387,"mutex_wait_us":143,"thread_start_us":2112,"threads_started":1}
I20260925 14:55:21.268443 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.247298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44362)
0925 14:55:21.248145 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:21.248185 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:21.248381 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:21.260864 (+ 12483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:21.260868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:21.260901 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:21.261326 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.261344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.264119 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.264129 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.265717 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.265730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.267658 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.267665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.267819 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.267859 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 14:55:21.267926 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:55:21.268058 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":3,"thread_start_us":12861,"threads_started":1}
I20260925 14:55:21.269477 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.247517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.17.133.193:41405)
0925 14:55:21.260196 (+ 12679us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.260204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.260230 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:55:21.260323 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.260331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.260637 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.261543 (+ 906us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:21.261782 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.263993 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.265880 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.267426 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.268869 (+ 1443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.269029 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.269104 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.269228 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12523,"thread_start_us":12375,"threads_started":1}
I20260925 14:55:21.271747 32061 tablet_service.cc:1467] Processing CreateTablet for tablet e8d430fed3c846fbb30b3ee6c67cee3f (DEFAULT_TABLE table=test-table [id=c0af5a5ad6f64d899d26138ee2460df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:21.272207 31927 tablet_service.cc:1467] Processing CreateTablet for tablet e8d430fed3c846fbb30b3ee6c67cee3f (DEFAULT_TABLE table=test-table [id=c0af5a5ad6f64d899d26138ee2460df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:21.273123 32061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d430fed3c846fbb30b3ee6c67cee3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:21.273511 31927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d430fed3c846fbb30b3ee6c67cee3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:21.281251 32277 tablet_bootstrap.cc:492] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: Bootstrap starting.
I20260925 14:55:21.284734 32277 tablet_bootstrap.cc:654] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:21.284837 32278 tablet_bootstrap.cc:492] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: Bootstrap starting.
I20260925 14:55:21.286082 32279 tablet_bootstrap.cc:492] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: Bootstrap starting.
I20260925 14:55:21.286130 32277 log.cc:824] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:21.288326 32278 tablet_bootstrap.cc:654] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:21.289280 32279 tablet_bootstrap.cc:654] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:21.289427 32278 log.cc:824] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:21.289963 32277 tablet_bootstrap.cc:492] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: No bootstrap required, opened a new log
I20260925 14:55:21.290280 32277 ts_tablet_manager.cc:1397] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260925 14:55:21.290428 32279 log.cc:824] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:21.292112 32278 tablet_bootstrap.cc:492] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: No bootstrap required, opened a new log
I20260925 14:55:21.292358 32278 ts_tablet_manager.cc:1397] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 14:55:21.293259 32279 tablet_bootstrap.cc:492] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: No bootstrap required, opened a new log
I20260925 14:55:21.293510 32279 ts_tablet_manager.cc:1397] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260925 14:55:21.299059 32278 raft_consensus.cc:348] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } }
I20260925 14:55:21.299468 32278 raft_consensus.cc:374] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:21.299597 32278 raft_consensus.cc:729] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cee73d048c4465a94507b827d256797, State: Initialized, Role: FOLLOWER
I20260925 14:55:21.300184 32278 consensus_queue.cc:260] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } }
I20260925 14:55:21.300254 32277 raft_consensus.cc:348] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } }
I20260925 14:55:21.300745 32277 raft_consensus.cc:374] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:21.300886 32277 raft_consensus.cc:729] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb67611044ac45ab8826731ed1522784, State: Initialized, Role: FOLLOWER
I20260925 14:55:21.301687 32278 ts_tablet_manager.cc:1428] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260925 14:55:21.302421 32277 consensus_queue.cc:260] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } }
I20260925 14:55:21.303479 32259 heartbeater.cc:503] Master 127.17.133.254:34843 was elected leader, sending a full tablet report...
I20260925 14:55:21.303254 32279 raft_consensus.cc:348] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } }
I20260925 14:55:21.303643 32279 raft_consensus.cc:374] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:21.303731 32279 raft_consensus.cc:729] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf0a857b7a24b13bc3fb4f4011a008a, State: Initialized, Role: FOLLOWER
I20260925 14:55:21.304069 32277 ts_tablet_manager.cc:1428] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260925 14:55:21.304332 32279 consensus_queue.cc:260] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } }
I20260925 14:55:21.305438 32126 heartbeater.cc:503] Master 127.17.133.254:34843 was elected leader, sending a full tablet report...
I20260925 14:55:21.306036 32279 ts_tablet_manager.cc:1428] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
W20260925 14:55:21.430148 32260 tablet.cc:2369] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:21.433962 32127 tablet.cc:2369] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:21.461872 31993 tablet.cc:2369] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:21.504242 32283 raft_consensus.cc:482] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:21.504608 32283 raft_consensus.cc:504] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } }
I20260925 14:55:21.506517 32283 leader_election.cc:288] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749), 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233)
I20260925 14:55:21.514473 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.506989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:42087 (local address 127.17.133.195:46749)
0925 14:55:21.507198 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.507204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.507995 (+ 791us) server_negotiation.cc:407] Connection header received
0925 14:55:21.508139 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.508145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.508223 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.508323 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:21.509193 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.510348 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.512699 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.513491 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.513644 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.513735 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.514083 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.514210 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:55:21.514514 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.507211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:44049 (local address 127.17.133.194:34233)
0925 14:55:21.507428 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.507434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.507963 (+ 529us) server_negotiation.cc:407] Connection header received
0925 14:55:21.508139 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.508145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.508223 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.508330 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:21.509193 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.510370 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.512746 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.513518 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.513698 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.513771 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.514126 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.514257 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:55:21.514658 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.506846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:34233 (local address 127.17.133.193:44049)
0925 14:55:21.507786 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:21.507807 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:21.507993 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:21.508352 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:21.508357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:21.508426 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:21.509019 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.509040 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.510530 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.510536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.512461 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.512475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.513773 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.513779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.513952 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.513976 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 14:55:21.514086 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:55:21.514209 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":227,"spinlock_wait_cycles":29440,"thread_start_us":530,"threads_started":1}
I20260925 14:55:21.514900 32287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.506897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:46749 (local address 127.17.133.193:42087)
0925 14:55:21.507865 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:21.507881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:21.508007 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:21.508352 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:21.508357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:21.508382 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:21.509046 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.509054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.510530 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.510536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.512423 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.512438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.513728 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.513734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.513929 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.513954 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 14:55:21.514028 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:55:21.514173 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":165,"spinlock_wait_cycles":896,"thread_start_us":716,"threads_started":1}
I20260925 14:55:21.515708 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d430fed3c846fbb30b3ee6c67cee3f" candidate_uuid: "2cee73d048c4465a94507b827d256797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" is_pre_election: true
I20260925 14:55:21.515805 32214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d430fed3c846fbb30b3ee6c67cee3f" candidate_uuid: "2cee73d048c4465a94507b827d256797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb67611044ac45ab8826731ed1522784" is_pre_election: true
I20260925 14:55:21.516317 32081 raft_consensus.cc:2454] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cee73d048c4465a94507b827d256797 in term 0.
I20260925 14:55:21.516317 32214 raft_consensus.cc:2454] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cee73d048c4465a94507b827d256797 in term 0.
I20260925 14:55:21.517078 31881 leader_election.cc:302] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2cee73d048c4465a94507b827d256797, 8bf0a857b7a24b13bc3fb4f4011a008a, bb67611044ac45ab8826731ed1522784; no voters:
I20260925 14:55:21.517536 32283 raft_consensus.cc:2781] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:21.517694 32283 raft_consensus.cc:482] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:21.517797 32283 raft_consensus.cc:3037] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:21.520224 32283 raft_consensus.cc:504] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } }
I20260925 14:55:21.520951 32283 leader_election.cc:288] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 1 election: Requested vote from peers bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749), 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233)
I20260925 14:55:21.521386 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d430fed3c846fbb30b3ee6c67cee3f" candidate_uuid: "2cee73d048c4465a94507b827d256797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a"
I20260925 14:55:21.521376 32214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d430fed3c846fbb30b3ee6c67cee3f" candidate_uuid: "2cee73d048c4465a94507b827d256797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb67611044ac45ab8826731ed1522784"
I20260925 14:55:21.521574 32214 raft_consensus.cc:3037] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:21.521592 32081 raft_consensus.cc:3037] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:21.523932 32214 raft_consensus.cc:2454] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cee73d048c4465a94507b827d256797 in term 1.
I20260925 14:55:21.523944 32081 raft_consensus.cc:2454] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cee73d048c4465a94507b827d256797 in term 1.
I20260925 14:55:21.524294 31880 leader_election.cc:302] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cee73d048c4465a94507b827d256797, bb67611044ac45ab8826731ed1522784; no voters:
I20260925 14:55:21.524609 32283 raft_consensus.cc:2781] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:21.525064 32283 raft_consensus.cc:686] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 1 LEADER]: Becoming Leader. State: Replica: 2cee73d048c4465a94507b827d256797, State: Running, Role: LEADER
I20260925 14:55:21.525610 32283 consensus_queue.cc:237] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } }
I20260925 14:55:21.529721 31802 catalog_manager.cc:5495] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cee73d048c4465a94507b827d256797 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "2cee73d048c4465a94507b827d256797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:21.551431 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.546596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.17.133.193:41405)
0925 14:55:21.546805 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.546810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.546826 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:55:21.546994 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.546997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.547075 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.547190 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:21.547508 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.548177 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.549541 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.549924 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.550011 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.550418 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:21.550916 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:21.551016 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.551090 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.551148 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:55:21.559595 32214 raft_consensus.cc:1261] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [term 1 FOLLOWER]: Refusing update from remote peer 2cee73d048c4465a94507b827d256797: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:21.559597 32081 raft_consensus.cc:1261] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [term 1 FOLLOWER]: Refusing update from remote peer 2cee73d048c4465a94507b827d256797: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:21.560545 32288 consensus_queue.cc:1035] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:21.560957 32283 consensus_queue.cc:1035] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:21.573122 32291 mvcc.cc:205] Tried to move back new op lower bound from 7333265905897676800 to 7333265905772269568. Current Snapshot: MvccSnapshot[applied={T|T < 7333265905897676800}]
I20260925 14:55:21.579834 32292 mvcc.cc:205] Tried to move back new op lower bound from 7333265905897676800 to 7333265905772269568. Current Snapshot: MvccSnapshot[applied={T|T < 7333265905897676800 or (T in {7333265905897676800})}]
I20260925 14:55:21.589072 31802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:21.589341 31802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:21.593287 31802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Sending DeleteTablet for 3 replicas of tablet e8d430fed3c846fbb30b3ee6c67cee3f
I20260925 14:55:21.594257 32194 tablet_service.cc:1514] Processing DeleteTablet for tablet e8d430fed3c846fbb30b3ee6c67cee3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:21 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:56210
I20260925 14:55:21.594292 31927 tablet_service.cc:1514] Processing DeleteTablet for tablet e8d430fed3c846fbb30b3ee6c67cee3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:21 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:44362
I20260925 14:55:21.596199 32061 tablet_service.cc:1514] Processing DeleteTablet for tablet e8d430fed3c846fbb30b3ee6c67cee3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:21 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:47822
I20260925 14:55:21.603649 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 31864
I20260925 14:55:21.606698 32302 tablet_replica.cc:330] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: stopping tablet replica
I20260925 14:55:21.607218 32302 raft_consensus.cc:2229] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:21.608062 32302 raft_consensus.cc:2258] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:21.611064 32302 ts_tablet_manager.cc:1905] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:21.616488 32303 tablet_replica.cc:330] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: stopping tablet replica
I20260925 14:55:21.617107 32303 raft_consensus.cc:2229] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:21.617800 32303 raft_consensus.cc:2258] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:21.618774 32302 ts_tablet_manager.cc:1918] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:21.620090 32302 log.cc:1196] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/e8d430fed3c846fbb30b3ee6c67cee3f
I20260925 14:55:21.620702 32302 ts_tablet_manager.cc:1939] T e8d430fed3c846fbb30b3ee6c67cee3f P 2cee73d048c4465a94507b827d256797: Deleting consensus metadata
I20260925 14:55:21.622380 31789 catalog_manager.cc:4841] TS 2cee73d048c4465a94507b827d256797 (127.17.133.193:41405): tablet e8d430fed3c846fbb30b3ee6c67cee3f (table test-table [id=c0af5a5ad6f64d899d26138ee2460df2]) successfully deleted
I20260925 14:55:21.623183 32301 tablet_replica.cc:330] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: stopping tablet replica
I20260925 14:55:21.623631 32301 raft_consensus.cc:2229] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:21.623697 32303 ts_tablet_manager.cc:1905] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:21.624116 32301 raft_consensus.cc:2258] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:21.626075 32301 ts_tablet_manager.cc:1905] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 14:55:21 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:55:21.630563 32303 ts_tablet_manager.cc:1918] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:21.630702 32303 log.cc:1196] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/e8d430fed3c846fbb30b3ee6c67cee3f
I20260925 14:55:21.631179 32303 ts_tablet_manager.cc:1939] T e8d430fed3c846fbb30b3ee6c67cee3f P 8bf0a857b7a24b13bc3fb4f4011a008a: Deleting consensus metadata
I20260925 14:55:21.632869 31788 catalog_manager.cc:4841] TS 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233): tablet e8d430fed3c846fbb30b3ee6c67cee3f (table test-table [id=c0af5a5ad6f64d899d26138ee2460df2]) successfully deleted
I20260925 14:55:21.633183 32301 ts_tablet_manager.cc:1918] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:21.633322 32301 log.cc:1196] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/e8d430fed3c846fbb30b3ee6c67cee3f
I20260925 14:55:21.633730 32301 ts_tablet_manager.cc:1939] T e8d430fed3c846fbb30b3ee6c67cee3f P bb67611044ac45ab8826731ed1522784: Deleting consensus metadata
I20260925 14:55:21.635282 31787 catalog_manager.cc:4841] TS bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749): tablet e8d430fed3c846fbb30b3ee6c67cee3f (table test-table [id=c0af5a5ad6f64d899d26138ee2460df2]) successfully deleted
I20260925 14:55:21.637508 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.604831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.17.133.193:41405)
0925 14:55:21.605010 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.605017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.613182 (+ 8165us) server_negotiation.cc:407] Connection header received
0925 14:55:21.613487 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.613495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.613600 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.613687 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:21.618452 (+ 4765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.619786 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.623452 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.624159 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.624278 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.630013 (+ 5735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:21.630026 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:21.630043 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:21.630099 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:21.634265 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.635139 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.635154 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.635156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.635295 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.635718 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.635724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.635728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.636212 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:21.636519 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.636856 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.637141 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:55:21.638175 31987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:21.734365 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 31864
I20260925 14:55:21.752787 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 31998
Sep 25 14:55:21 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:55:21.767598 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.753613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.17.133.194:34233)
0925 14:55:21.753817 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.753822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.753839 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:55:21.753893 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.753897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.753956 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.754047 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:21.755635 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.756449 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.757947 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.758335 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.758438 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.762447 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:21.762462 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:21.762478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:21.762507 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:21.765195 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.765825 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.765832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.765834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.765947 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.766263 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.766266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.766268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.766621 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:21.766835 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.767077 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.767229 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:55:21.768298 32121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:21.855392 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 31998
I20260925 14:55:21.871841 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 32131
Sep 25 14:55:21 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:55:21.886929 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.872591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.17.133.195:46749)
0925 14:55:21.872822 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.872830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.872850 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:55:21.873045 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.873051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.873127 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.873237 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:21.874528 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.875234 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.876477 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.877024 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.877156 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.881536 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:21.881559 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:21.881575 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:21.881604 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:21.884100 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.884848 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.884859 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.884867 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.885010 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.885405 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.885410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.885413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.885759 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:21.886077 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.886390 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.886594 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:55:21.887588 32254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:21.976141 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 32131
I20260925 14:55:21.991995 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 31771
I20260925 14:55:21.993098 31833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:22.099419 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 31771
I20260925 14:55:22.119977 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34843
--webserver_interface=127.17.133.254
--webserver_port=35993
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:22.436826 32309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:22.437126 32309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:22.437285 32309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:22.444108 32309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:22.444199 32309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:22.444276 32309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:22.444358 32309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:22.444418 32309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:22.453629 32309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=35993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:22.454854 32309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:22.456101 32309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:22.476824 32316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:22.476869 32315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:22.476897 32318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:22.477687 32309 server_base.cc:1034] running on GCE node
I20260925 14:55:22.478564 32309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:22.479983 32309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:22.481179 32309 hybrid_clock.cc:648] HybridClock initialized: now 1790348122481139 us; error 37 us; skew 500 ppm
I20260925 14:55:22.481566 32309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:22 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348122, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:22.485133 32309 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:55:22.486788 32309 webserver.cc:460] Webserver started at http://127.17.133.254:35993/ using document root <none> and password file <none>
I20260925 14:55:22.487550 32309 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:22.487708 32309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:22.493456 32309 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 14:55:22.496402 32325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:22.497732 32309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:55:22.497917 32309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "c9f316ce510d4ea0a10a92384d4b0d80"
format_stamp: "Formatted at 2026-09-25 14:55:19 on dist-test-slave-v5vl"
server_key: "93530d2e1fb257f80196efc1039f07a4"
server_key_iv: "da66193076bba194290c3d53bc4c7835"
server_key_version: "encryptionkey@0"
I20260925 14:55:22.498986 32309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:22.525761 32309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:22.530403 32309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:22.530694 32309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:22.548987 32309 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:34843
I20260925 14:55:22.549023 32376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:34843 every 8 connection(s)
I20260925 14:55:22.550402 32309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:22.556913 32377 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:55:22.559839 32377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Bootstrap starting.
I20260925 14:55:22.560935 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 32309
I20260925 14:55:22.561939 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:41405
--local_ip_for_outbound_sockets=127.17.133.193
--tserver_master_addrs=127.17.133.254:34843
--webserver_port=33279
--webserver_interface=127.17.133.193
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:22.582405 32377 log.cc:824] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:22.594835 32377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:55:22.595427 32377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Bootstrap complete.
I20260925 14:55:22.605033 32377 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:22.605904 32377 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f316ce510d4ea0a10a92384d4b0d80, State: Initialized, Role: FOLLOWER
I20260925 14:55:22.606479 32377 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:22.606721 32377 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:22.606853 32377 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:22.607026 32377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:55:22.611334 32377 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:22.611886 32377 leader_election.cc:302] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9f316ce510d4ea0a10a92384d4b0d80; no voters:
I20260925 14:55:22.612485 32377 leader_election.cc:288] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:55:22.612637 32381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:55:22.614929 32377 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:22.619751 32381 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 2 LEADER]: Becoming Leader. State: Replica: c9f316ce510d4ea0a10a92384d4b0d80, State: Running, Role: LEADER
I20260925 14:55:22.620419 32381 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:22.623169 32382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } } }
I20260925 14:55:22.623632 32382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:22.630434 32383 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9f316ce510d4ea0a10a92384d4b0d80. Latest consensus state: current_term: 2 leader_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } } }
I20260925 14:55:22.630828 32383 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:22.631464 32392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:22.641355 32392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:22.643100 32392 catalog_manager.cc:1259] Loaded cluster ID: ceab647ffa544bb08e4ed9c8d3b8c539
I20260925 14:55:22.643216 32392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:22.646962 32392 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 14:55:22.650676 32399 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:55:22.659991 32392 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 14:55:22.666585 32392 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:22.667825 32392 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Loaded TSK: 0
I20260925 14:55:22.668960 32392 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:55:22.915930 32379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:22.916209 32379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:22.916368 32379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:22.923671 32379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:22.923801 32379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:22.924005 32379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:55:22.933647 32379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:41405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=33279
--tserver_master_addrs=127.17.133.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:22.934958 32379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:22.936537 32379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:22.950028 32405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:22.949994 32406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:22.951191 32408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:22.951267 32379 server_base.cc:1034] running on GCE node
I20260925 14:55:22.951942 32379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:22.952920 32379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:22.954128 32379 hybrid_clock.cc:648] HybridClock initialized: now 1790348122954074 us; error 51 us; skew 500 ppm
I20260925 14:55:22.954459 32379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:22 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348122, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:22.960534 32379 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:55:22.962159 32379 webserver.cc:460] Webserver started at http://127.17.133.193:33279/ using document root <none> and password file <none>
I20260925 14:55:22.962832 32379 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:22.962972 32379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:22.968252 32379 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:22.970933 32415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:22.972239 32379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:55:22.972393 32379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "2cee73d048c4465a94507b827d256797"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "ba1f4bcd96ae8ea60c2ac2493ba6ef73"
server_key_iv: "9b28511edb31e21a536412b265efe175"
server_key_version: "encryptionkey@0"
I20260925 14:55:22.973316 32379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:22.996793 32379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:23.000842 32379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:23.001080 32379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:23.002074 32379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:23.003458 32379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:23.003583 32379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.003705 32379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:23.003786 32379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.037120 32379 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:41405
I20260925 14:55:23.037207 32527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:41405 every 8 connection(s)
I20260925 14:55:23.038894 32379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:23.046597 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 32379
I20260925 14:55:23.047367 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:34233
--local_ip_for_outbound_sockets=127.17.133.194
--tserver_master_addrs=127.17.133.254:34843
--webserver_port=38207
--webserver_interface=127.17.133.194
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:23 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348122, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:23.065410 32531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.042366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:48839)
0925 14:55:23.043190 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:23.043247 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:23.045077 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:23.046468 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:23.046481 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:23.047225 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:23.048063 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.048091 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.049774 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.049782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.051975 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.051993 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.053058 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.053066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.053375 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.054109 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:23.054142 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:23.056676 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:23.061307 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:23.061314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:23.061334 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:23.061783 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:23.062262 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:23.062265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:23.062267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:23.062442 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:23.063351 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:23.063356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:23.063652 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 14:55:23.063888 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 14:55:23.064283 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":367,"threads_started":1}
I20260925 14:55:23.065893 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.042107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:48839 (local address 127.17.133.254:34843)
0925 14:55:23.042743 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.042762 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.043375 (+ 613us) server_negotiation.cc:407] Connection header received
0925 14:55:23.045584 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.045597 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.045928 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.046222 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:23.048295 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.049593 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.052145 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.052842 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.052964 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.056949 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:23.056961 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:23.056981 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:23.057031 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:23.061147 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.061921 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.061932 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.061942 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.062146 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.062550 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.062558 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.062564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.063216 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:23.063406 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.064148 (+ 742us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.064645 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":281,"threads_started":1}
I20260925 14:55:23.068817 32528 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34843
I20260925 14:55:23.069135 32528 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:23.069885 32528 heartbeater.cc:511] Master 127.17.133.254:34843 requested a full tablet report, sending...
I20260925 14:55:23.072083 32342 ts_manager.cc:194] Registered new tserver with Master: 2cee73d048c4465a94507b827d256797 (127.17.133.193:41405)
I20260925 14:55:23.073639 32342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:48839
W20260925 14:55:23.363480 32532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:23.363713 32532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:23.363849 32532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:23.370609 32532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:23.370695 32532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:23.370864 32532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:55:23.379966 32532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:34233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=38207
--tserver_master_addrs=127.17.133.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:23.381174 32532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:23.382463 32532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:23.395081 32537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:23.395032 32540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:23.395118 32538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:23.395855 32532 server_base.cc:1034] running on GCE node
I20260925 14:55:23.396335 32532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:23.397243 32532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:23.398460 32532 hybrid_clock.cc:648] HybridClock initialized: now 1790348123398425 us; error 47 us; skew 500 ppm
I20260925 14:55:23.398762 32532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:23 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:23.401832 32532 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:55:23.403219 32532 webserver.cc:460] Webserver started at http://127.17.133.194:38207/ using document root <none> and password file <none>
I20260925 14:55:23.403824 32532 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:23.403968 32532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:23.408977 32532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:55:23.411623 32547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.412876 32532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:23.413034 32532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "8bf0a857b7a24b13bc3fb4f4011a008a"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "016aece6c0dd93f9b8208389bbe39cc5"
server_key_iv: "bbac9f056e2738662d23dc9591f1df20"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.413923 32532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:23.440850 32532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:23.444458 32532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:23.444772 32532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:23.445709 32532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:23.446924 32532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:23.447104 32532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.447247 32532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:23.447330 32532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.479156 32532 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:34233
I20260925 14:55:23.479208 32659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:34233 every 8 connection(s)
I20260925 14:55:23.480615 32532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:23.482643 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 32532
I20260925 14:55:23.483409 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:46749
--local_ip_for_outbound_sockets=127.17.133.195
--tserver_master_addrs=127.17.133.254:34843
--webserver_port=35661
--webserver_interface=127.17.133.195
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:23 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:23.500504 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.483181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:60235 (local address 127.17.133.254:34843)
0925 14:55:23.483346 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.483351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.484792 (+ 1441us) server_negotiation.cc:407] Connection header received
0925 14:55:23.486569 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.486574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.486665 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.486773 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:23.488215 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.488990 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.490487 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.490881 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.491001 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.494558 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:23.494562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:23.494566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:23.494597 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:23.497111 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.498060 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.498064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.498066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.498177 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.498688 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.498691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.498693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.498995 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:23.499136 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.500097 (+ 961us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.500273 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:55:23.502313 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.483878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:60235)
0925 14:55:23.484641 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:23.484683 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:23.486301 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:23.486999 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:23.487006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:23.487478 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:23.488054 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.488075 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.489137 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.489143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.490325 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.490336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.491075 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.491082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.491286 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.491905 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:23.491925 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:23.494316 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:23.497269 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:23.497282 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:23.497300 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:23.497941 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:23.498303 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:23.498311 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:23.498317 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:23.498569 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:23.499153 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:23.499174 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:23.499670 (+ 496us) client_negotiation.cc:769] Sending connection context
0925 14:55:23.500015 (+ 345us) client_negotiation.cc:240] Negotiation successful
0925 14:55:23.500691 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":296,"threads_started":1}
I20260925 14:55:23.503885 32660 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34843
I20260925 14:55:23.504195 32660 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:23.504949 32660 heartbeater.cc:511] Master 127.17.133.254:34843 requested a full tablet report, sending...
I20260925 14:55:23.506397 32342 ts_manager.cc:194] Registered new tserver with Master: 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233)
I20260925 14:55:23.507164 32342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:60235
W20260925 14:55:23.806246 32663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:23.806571 32663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:23.806741 32663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:23.814134 32663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:23.814270 32663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:23.814456 32663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:55:23.823952 32663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:46749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=35661
--tserver_master_addrs=127.17.133.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:23.825212 32663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:23.826468 32663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:23.838483 32671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:23.838562 32668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:23.838593 32669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:23.839401 32663 server_base.cc:1034] running on GCE node
I20260925 14:55:23.839887 32663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:23.840669 32663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:23.841861 32663 hybrid_clock.cc:648] HybridClock initialized: now 1790348123841807 us; error 53 us; skew 500 ppm
I20260925 14:55:23.842096 32663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:23 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:23.845037 32663 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:55:23.846313 32663 webserver.cc:460] Webserver started at http://127.17.133.195:35661/ using document root <none> and password file <none>
I20260925 14:55:23.846861 32663 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:23.846969 32663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:23.851799 32663 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:55:23.854156 32678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.855374 32663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:23.855538 32663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "bb67611044ac45ab8826731ed1522784"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-v5vl"
server_key: "a4a5157e042de1a84367eac05db83165"
server_key_iv: "3cfad6ec043f2a189d9b3908f9e6d1b6"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.856419 32663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:23.871719 32663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:23.875379 32663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:23.875641 32663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:23.876662 32663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:23.878015 32663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:23.878134 32663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.878268 32663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:23.878376 32663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.909418 32663 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:46749
I20260925 14:55:23.909482 322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:46749 every 8 connection(s)
I20260925 14:55:23.911013 32663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:23.916778 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 32663
Sep 25 14:55:23 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:23.933652 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.913575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:44051 (local address 127.17.133.254:34843)
0925 14:55:23.913792 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.913799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.914956 (+ 1157us) server_negotiation.cc:407] Connection header received
0925 14:55:23.916950 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.916958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.917059 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.917178 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:23.919080 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.920215 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.922011 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.922665 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.922747 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.926820 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:23.926825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:23.926828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:23.926858 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:23.930263 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.931078 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.931085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.931093 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.931249 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.931665 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.931670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.931673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.932018 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:23.932161 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.933069 (+ 908us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.933291 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:55:23.934430 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.917744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.17.133.254:34843)
0925 14:55:23.918300 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.918306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.919582 (+ 1276us) server_negotiation.cc:407] Connection header received
0925 14:55:23.919732 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.919736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.919791 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.919875 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:23.921547 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.922495 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.924426 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.924914 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.925002 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.927847 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:23.927853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:23.927857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:23.927925 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:23.931313 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.932097 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.932102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.932106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.932264 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.932642 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.932650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.932656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.932991 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:23.933210 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.933534 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.934058 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":287,"threads_started":1}
I20260925 14:55:23.934954 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.914105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:44051)
0925 14:55:23.914791 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:23.914834 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:23.916641 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:23.917467 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:23.917476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:23.918195 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:23.918907 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.918925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.920360 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.920365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.921875 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.921888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.922875 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.922883 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.923211 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.923942 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:23.923974 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:23.926598 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:23.930442 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:23.930452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:23.930471 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:23.930944 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:23.931359 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:23.931363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:23.931365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:23.931542 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:23.932145 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:23.932155 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:23.932628 (+ 473us) client_negotiation.cc:769] Sending connection context
0925 14:55:23.932947 (+ 319us) client_negotiation.cc:240] Negotiation successful
0925 14:55:23.933521 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":272,"threads_started":1}
I20260925 14:55:23.936261 323 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34843
I20260925 14:55:23.936632 323 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:23.937412 323 heartbeater.cc:511] Master 127.17.133.254:34843 requested a full tablet report, sending...
I20260925 14:55:23.938712 32342 ts_manager.cc:194] Registered new tserver with Master: bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749)
I20260925 14:55:23.939520 32342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:44051
I20260925 14:55:23.949493 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:23.956236 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.950443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.17.133.254:34843)
0925 14:55:23.950608 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.950612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.951087 (+ 475us) server_negotiation.cc:407] Connection header received
0925 14:55:23.951258 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.951262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.951400 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.951713 (+ 313us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:23.952093 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.952885 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.954317 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.954706 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.954799 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.955231 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:23.955791 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:23.955902 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.955981 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.956045 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:55:23.957329 32342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:23.964768 32342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:23.996637 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.976010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.17.133.194:34233)
0925 14:55:23.985926 (+ 9916us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.985938 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.987484 (+ 1546us) server_negotiation.cc:407] Connection header received
0925 14:55:23.987601 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.987608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.987883 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.989088 (+ 1205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:23.989407 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.990977 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.993838 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.995377 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.995587 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.995764 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.996077 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.996245 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9708,"thread_start_us":9524,"threads_started":1}
I20260925 14:55:23.996641 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.975813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:34233 (local address 127.0.0.1:47840)
0925 14:55:23.986422 (+ 10609us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:23.986474 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:23.986666 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:23.988160 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:23.988169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:23.988208 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:23.988725 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.988743 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.991128 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.991134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.993641 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.993664 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.995606 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.995612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.995819 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.995870 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 14:55:23.996020 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 14:55:23.996222 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10322,"thread_start_us":10150,"threads_started":1}
I20260925 14:55:23.997361 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.976443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44388)
0925 14:55:23.989278 (+ 12835us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:23.989294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:23.989413 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:23.990682 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:23.990686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:23.990708 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:23.991003 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.991011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.992486 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.992493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.994451 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.994461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.996005 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.996010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.996293 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.996454 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 14:55:23.996540 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:55:23.996945 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12704,"mutex_wait_us":373,"thread_start_us":12570,"threads_started":1}
I20260925 14:55:23.997570 337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.976726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.17.133.195:46749)
0925 14:55:23.988550 (+ 11824us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.988558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.988627 (+ 69us) server_negotiation.cc:407] Connection header received
0925 14:55:23.988760 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.988770 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.989114 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.989312 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:23.989974 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.991490 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.994084 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.995392 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.995585 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.995764 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.997060 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.997220 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11645,"thread_start_us":11480,"threads_started":1}
I20260925 14:55:23.998009 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.977231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.17.133.193:41405)
0925 14:55:23.988022 (+ 10791us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.988033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.989547 (+ 1514us) server_negotiation.cc:407] Connection header received
0925 14:55:23.989676 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.989683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.990261 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.990464 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:23.991208 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.992339 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.994595 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.995617 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.995774 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.995941 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.996676 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.997702 (+ 1026us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10613,"thread_start_us":10526,"threads_started":1}
I20260925 14:55:23.998107 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.976049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:46749 (local address 127.0.0.1:56238)
0925 14:55:23.987137 (+ 11088us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:23.987152 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:23.987279 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:23.989325 (+ 2046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:23.989330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:23.989357 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:23.989738 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.989751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.991656 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.991661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.993909 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.993924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.995986 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.995994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.996297 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.996452 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 14:55:23.996540 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:55:23.996709 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10943,"mutex_wait_us":144,"thread_start_us":10794,"threads_started":1}
I20260925 14:55:24.001011 32726 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9961a4de5f43209b83a313a2c0b062 (DEFAULT_TABLE table=test-table [id=7ffb676976be46539eb40e9d351c333d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:24.001055 32595 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9961a4de5f43209b83a313a2c0b062 (DEFAULT_TABLE table=test-table [id=7ffb676976be46539eb40e9d351c333d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:24.001430 32463 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9961a4de5f43209b83a313a2c0b062 (DEFAULT_TABLE table=test-table [id=7ffb676976be46539eb40e9d351c333d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:24.002447 32595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9961a4de5f43209b83a313a2c0b062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:24.002472 32726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9961a4de5f43209b83a313a2c0b062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:24.003926 32463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9961a4de5f43209b83a313a2c0b062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:24.014285 339 tablet_bootstrap.cc:492] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: Bootstrap starting.
I20260925 14:55:24.018023 339 tablet_bootstrap.cc:654] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:24.019202 339 log.cc:824] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:24.021556 341 tablet_bootstrap.cc:492] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Bootstrap starting.
I20260925 14:55:24.022382 339 tablet_bootstrap.cc:492] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: No bootstrap required, opened a new log
I20260925 14:55:24.022660 339 ts_tablet_manager.cc:1397] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 14:55:24.024487 341 tablet_bootstrap.cc:654] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:24.025821 341 log.cc:824] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:24.026758 340 tablet_bootstrap.cc:492] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: Bootstrap starting.
I20260925 14:55:24.029093 341 tablet_bootstrap.cc:492] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: No bootstrap required, opened a new log
I20260925 14:55:24.029394 341 ts_tablet_manager.cc:1397] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 14:55:24.030265 340 tablet_bootstrap.cc:654] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:24.030821 339 raft_consensus.cc:348] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:24.031210 339 raft_consensus.cc:374] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:24.031337 339 raft_consensus.cc:729] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb67611044ac45ab8826731ed1522784, State: Initialized, Role: FOLLOWER
I20260925 14:55:24.031517 340 log.cc:824] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:24.032358 339 consensus_queue.cc:260] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:24.035089 340 tablet_bootstrap.cc:492] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: No bootstrap required, opened a new log
I20260925 14:55:24.035403 340 ts_tablet_manager.cc:1397] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260925 14:55:24.035939 323 heartbeater.cc:503] Master 127.17.133.254:34843 was elected leader, sending a full tablet report...
I20260925 14:55:24.036818 339 ts_tablet_manager.cc:1428] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260925 14:55:24.039316 341 raft_consensus.cc:348] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:24.039798 341 raft_consensus.cc:374] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:24.039908 341 raft_consensus.cc:729] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cee73d048c4465a94507b827d256797, State: Initialized, Role: FOLLOWER
I20260925 14:55:24.040575 341 consensus_queue.cc:260] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:24.047776 340 raft_consensus.cc:348] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:24.048254 340 raft_consensus.cc:374] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:24.048345 340 raft_consensus.cc:729] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf0a857b7a24b13bc3fb4f4011a008a, State: Initialized, Role: FOLLOWER
I20260925 14:55:24.048925 32528 heartbeater.cc:503] Master 127.17.133.254:34843 was elected leader, sending a full tablet report...
I20260925 14:55:24.049589 341 ts_tablet_manager.cc:1428] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260925 14:55:24.049081 340 consensus_queue.cc:260] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:24.050896 32660 heartbeater.cc:503] Master 127.17.133.254:34843 was elected leader, sending a full tablet report...
I20260925 14:55:24.051394 340 ts_tablet_manager.cc:1428] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260925 14:55:24.091274 347 raft_consensus.cc:482] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:24.091542 347 raft_consensus.cc:504] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:24.093333 347 leader_election.cc:288] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749), 2cee73d048c4465a94507b827d256797 (127.17.133.193:41405)
I20260925 14:55:24.099905 337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.093743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:51441 (local address 127.17.133.195:46749)
0925 14:55:24.093994 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.094002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.094907 (+ 905us) server_negotiation.cc:407] Connection header received
0925 14:55:24.095059 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.095064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.095131 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.095198 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:24.095709 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.096441 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.098180 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.098979 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.099160 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.099238 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.099507 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.099626 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 14:55:24.100154 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.093933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:47399 (local address 127.17.133.193:41405)
0925 14:55:24.094138 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.094146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.094826 (+ 680us) server_negotiation.cc:407] Connection header received
0925 14:55:24.095351 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.095356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.095422 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.095524 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:24.096059 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.096820 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.098491 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.099251 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.099433 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.099507 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.099784 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.099917 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:55:24.100037 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.093801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:46749 (local address 127.17.133.194:51441)
0925 14:55:24.094815 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.094829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:24.094936 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:24.095240 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:24.095244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:24.095261 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:24.095551 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:24.095559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.096575 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.096607 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:24.098031 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:24.098042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.099200 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.099206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:24.099349 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.099371 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:55:24.099484 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:55:24.099619 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":34,"thread_start_us":528,"threads_started":1}
I20260925 14:55:24.100550 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.093801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.17.133.194:47399)
0925 14:55:24.094710 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.094726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:24.094910 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:24.095546 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:24.095550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:24.095573 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:24.095924 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:24.095931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.096949 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.096954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:24.098327 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:24.098338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.099463 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.099469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:24.099632 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.099654 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:55:24.099760 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:55:24.099905 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":1,"thread_start_us":502,"threads_started":1}
I20260925 14:55:24.101289 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9961a4de5f43209b83a313a2c0b062" candidate_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb67611044ac45ab8826731ed1522784" is_pre_election: true
I20260925 14:55:24.101553 32483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9961a4de5f43209b83a313a2c0b062" candidate_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cee73d048c4465a94507b827d256797" is_pre_election: true
I20260925 14:55:24.101784 32746 raft_consensus.cc:2454] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf0a857b7a24b13bc3fb4f4011a008a in term 0.
I20260925 14:55:24.102108 32483 raft_consensus.cc:2454] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf0a857b7a24b13bc3fb4f4011a008a in term 0.
I20260925 14:55:24.102607 32548 leader_election.cc:302] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf0a857b7a24b13bc3fb4f4011a008a, bb67611044ac45ab8826731ed1522784; no voters:
I20260925 14:55:24.102993 347 raft_consensus.cc:2781] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:24.103128 347 raft_consensus.cc:482] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:24.103246 347 raft_consensus.cc:3037] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:24.105461 347 raft_consensus.cc:504] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:24.106066 347 leader_election.cc:288] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [CANDIDATE]: Term 1 election: Requested vote from peers bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749), 2cee73d048c4465a94507b827d256797 (127.17.133.193:41405)
I20260925 14:55:24.106500 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9961a4de5f43209b83a313a2c0b062" candidate_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb67611044ac45ab8826731ed1522784"
I20260925 14:55:24.106528 32483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9961a4de5f43209b83a313a2c0b062" candidate_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cee73d048c4465a94507b827d256797"
I20260925 14:55:24.106689 32746 raft_consensus.cc:3037] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:24.106699 32483 raft_consensus.cc:3037] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:24.108831 32483 raft_consensus.cc:2454] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf0a857b7a24b13bc3fb4f4011a008a in term 1.
I20260925 14:55:24.108865 32746 raft_consensus.cc:2454] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf0a857b7a24b13bc3fb4f4011a008a in term 1.
I20260925 14:55:24.109249 32550 leader_election.cc:302] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cee73d048c4465a94507b827d256797, 8bf0a857b7a24b13bc3fb4f4011a008a; no voters:
I20260925 14:55:24.109543 347 raft_consensus.cc:2781] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:24.110037 347 raft_consensus.cc:686] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 1 LEADER]: Becoming Leader. State: Replica: 8bf0a857b7a24b13bc3fb4f4011a008a, State: Running, Role: LEADER
I20260925 14:55:24.110561 347 consensus_queue.cc:237] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:24.115366 32341 catalog_manager.cc:5495] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194). New cstate: current_term: 1 leader_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:24.153565 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.148265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.17.133.194:34233)
0925 14:55:24.148499 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.148505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.148524 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:55:24.148688 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.148692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.148810 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.148969 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:24.149361 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.150063 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.151416 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.151816 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.151949 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.152433 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:24.153012 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:24.153167 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.153220 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.153310 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:55:24.162058 32483 raft_consensus.cc:1261] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Refusing update from remote peer 8bf0a857b7a24b13bc3fb4f4011a008a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:24.162057 32746 raft_consensus.cc:1261] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [term 1 FOLLOWER]: Refusing update from remote peer 8bf0a857b7a24b13bc3fb4f4011a008a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 14:55:24.162848 324 tablet.cc:2369] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:24.163138 347 consensus_queue.cc:1035] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:24.163594 350 consensus_queue.cc:1035] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:24.191634 32342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:24.191916 32342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:24.195174 32342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Sending DeleteTablet for 3 replicas of tablet 5b9961a4de5f43209b83a313a2c0b062
I20260925 14:55:24.196346 32595 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b9961a4de5f43209b83a313a2c0b062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:24 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:47840
I20260925 14:55:24.196341 32726 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b9961a4de5f43209b83a313a2c0b062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:24 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:56238
I20260925 14:55:24.196529 32463 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b9961a4de5f43209b83a313a2c0b062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:24 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:44388
I20260925 14:55:24.197417 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 32379
I20260925 14:55:24.210640 365 tablet_replica.cc:330] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: stopping tablet replica
I20260925 14:55:24.210896 364 tablet_replica.cc:330] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: stopping tablet replica
I20260925 14:55:24.211436 363 tablet_replica.cc:330] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: stopping tablet replica
I20260925 14:55:24.211730 364 raft_consensus.cc:2229] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:24.211931 363 raft_consensus.cc:2229] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:24.212514 363 raft_consensus.cc:2258] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:24.212683 364 raft_consensus.cc:2258] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:24.213896 365 raft_consensus.cc:2229] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:24.214884 364 ts_tablet_manager.cc:1905] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:24.215440 365 raft_consensus.cc:2258] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:24.217314 363 ts_tablet_manager.cc:1905] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:24.218748 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.198360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.17.133.193:41405)
0925 14:55:24.198536 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.198542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.198560 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:55:24.198612 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.198618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.198702 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.198801 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:24.200924 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.202131 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.204619 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.205500 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.205662 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.210156 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:24.210179 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:24.210201 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:24.210251 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:24.214170 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.215193 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.215200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.215203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.215354 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.215849 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.215854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.215857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.216370 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:24.217952 (+ 1582us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.218107 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.218348 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:55:24.219280 365 ts_tablet_manager.cc:1905] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:24.219447 32523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:24.222749 364 ts_tablet_manager.cc:1918] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:24.222914 364 log.cc:1196] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/5b9961a4de5f43209b83a313a2c0b062
I20260925 14:55:24.223512 364 ts_tablet_manager.cc:1939] T 5b9961a4de5f43209b83a313a2c0b062 P 8bf0a857b7a24b13bc3fb4f4011a008a: Deleting consensus metadata
I20260925 14:55:24.225380 32327 catalog_manager.cc:4841] TS 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233): tablet 5b9961a4de5f43209b83a313a2c0b062 (table test-table [id=7ffb676976be46539eb40e9d351c333d]) successfully deleted
I20260925 14:55:24.227784 363 ts_tablet_manager.cc:1918] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:24.228214 363 log.cc:1196] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/5b9961a4de5f43209b83a313a2c0b062
I20260925 14:55:24.229410 363 ts_tablet_manager.cc:1939] T 5b9961a4de5f43209b83a313a2c0b062 P bb67611044ac45ab8826731ed1522784: Deleting consensus metadata
I20260925 14:55:24.231153 32326 catalog_manager.cc:4841] TS bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749): tablet 5b9961a4de5f43209b83a313a2c0b062 (table test-table [id=7ffb676976be46539eb40e9d351c333d]) successfully deleted
I20260925 14:55:24.335677 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 32379
W20260925 14:55:24.352502 32328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.133.193:41405 (error 108)
I20260925 14:55:24.352746 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 32532
I20260925 14:55:24.354756 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.354195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44404)
0925 14:55:24.354471 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.354584 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:41405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260925 14:55:24.355000 32328 catalog_manager.cc:4584] TS 2cee73d048c4465a94507b827d256797 (127.17.133.193:41405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b9961a4de5f43209b83a313a2c0b062: Network error: Client connection negotiation failed: client connection to 127.17.133.193:41405: connect: Connection refused (error 111)
I20260925 14:55:24.367280 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.353539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.17.133.194:34233)
0925 14:55:24.353742 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.353748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.353894 (+ 146us) server_negotiation.cc:407] Connection header received
0925 14:55:24.354067 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.354071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.354134 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.354257 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:24.355760 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.356545 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.357909 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.358363 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.358492 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.361273 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:24.361286 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:24.361300 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:24.361335 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:24.364287 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.365150 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.365157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.365159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.365296 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.365703 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.365716 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.365718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.366086 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:24.366409 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.366704 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.366939 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:55:24.367997 32655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:24.408670 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.408153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44414)
0925 14:55:24.408356 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.408466 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:41405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 14:55:24.409808 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.409409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44426)
0925 14:55:24.409546 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.409660 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:41405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 14:55:24.470345 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.469848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44440)
0925 14:55:24.470073 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.470167 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:41405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 14:55:24.471417 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.471087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44454)
0925 14:55:24.471232 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.471305 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:41405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 14:55:24.484879 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 32532
I20260925 14:55:24.502624 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 32663
I20260925 14:55:24.515609 337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.503379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.17.133.195:46749)
0925 14:55:24.503548 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.503552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.503584 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:55:24.503788 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.503792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.503851 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.503931 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:24.505714 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.506391 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.507561 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.507900 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.507980 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.510655 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:24.510666 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:24.510679 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:24.510720 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:24.513292 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.513883 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.513890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.513892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.514019 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.514315 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.514318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.514320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.514631 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:24.514831 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.515063 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.515239 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:55:24.516224 318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:24.552222 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.551702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44468)
0925 14:55:24.551946 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.552049 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:41405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 14:55:24.553476 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.553042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44480)
0925 14:55:24.553252 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.553331 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:41405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 14:55:24.604116 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 32663
I20260925 14:55:24.621459 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 32309
I20260925 14:55:24.622521 32372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:24.732712 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.732097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:41405 (local address 127.0.0.1:44492)
0925 14:55:24.732313 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.732428 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:41405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 14:55:24.732872 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 32309
I20260925 14:55:24.754535 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34843
--webserver_interface=127.17.133.254
--webserver_port=35993
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:25.074136 371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:25.074468 371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:25.074642 371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:25.081642 371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:25.081770 371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:25.081874 371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:25.081975 371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:25.082062 371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:25.091096 371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=35993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:25.092231 371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:25.093467 371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:25.104308 376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:25.104324 377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:25.104418 379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:25.105281 371 server_base.cc:1034] running on GCE node
I20260925 14:55:25.105983 371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:25.107244 371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:25.108457 371 hybrid_clock.cc:648] HybridClock initialized: now 1790348125108416 us; error 55 us; skew 500 ppm
I20260925 14:55:25.108816 371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:25 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:25.111662 371 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:55:25.113127 371 webserver.cc:460] Webserver started at http://127.17.133.254:35993/ using document root <none> and password file <none>
I20260925 14:55:25.113749 371 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:25.113879 371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:25.118844 371 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 14:55:25.121505 386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:25.122709 371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:55:25.122876 371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "c9f316ce510d4ea0a10a92384d4b0d80"
format_stamp: "Formatted at 2026-09-25 14:55:19 on dist-test-slave-v5vl"
server_key: "93530d2e1fb257f80196efc1039f07a4"
server_key_iv: "da66193076bba194290c3d53bc4c7835"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.123799 371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:25.143198 371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:25.147282 371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:25.147589 371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:25.166085 371 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:34843
I20260925 14:55:25.166107 437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:34843 every 8 connection(s)
I20260925 14:55:25.167635 371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:25.174551 438 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:55:25.175323 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 371
I20260925 14:55:25.176306 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:41405
--local_ip_for_outbound_sockets=127.17.133.193
--tserver_master_addrs=127.17.133.254:34843
--webserver_port=33279
--webserver_interface=127.17.133.193
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:25.179482 438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Bootstrap starting.
I20260925 14:55:25.200284 438 log.cc:824] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:25.215426 438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:55:25.215920 438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80: Bootstrap complete.
I20260925 14:55:25.224881 438 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:25.225796 438 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f316ce510d4ea0a10a92384d4b0d80, State: Initialized, Role: FOLLOWER
I20260925 14:55:25.226413 438 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:25.226634 438 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:25.226742 438 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:25.226936 438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:55:25.230620 438 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:25.231109 438 leader_election.cc:302] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9f316ce510d4ea0a10a92384d4b0d80; no voters:
I20260925 14:55:25.231740 438 leader_election.cc:288] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:55:25.231876 442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:55:25.233923 438 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:25.238974 442 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [term 3 LEADER]: Becoming Leader. State: Replica: c9f316ce510d4ea0a10a92384d4b0d80, State: Running, Role: LEADER
I20260925 14:55:25.239606 442 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } }
I20260925 14:55:25.242316 443 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } } }
I20260925 14:55:25.242697 443 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:25.243316 444 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9f316ce510d4ea0a10a92384d4b0d80. Latest consensus state: current_term: 3 leader_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f316ce510d4ea0a10a92384d4b0d80" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34843 } } }
I20260925 14:55:25.243610 444 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f316ce510d4ea0a10a92384d4b0d80 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:25.265787 452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:25.283551 452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:25.285368 452 catalog_manager.cc:1259] Loaded cluster ID: ceab647ffa544bb08e4ed9c8d3b8c539
I20260925 14:55:25.285492 452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 14:55:25.288060 452 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
I20260925 14:55:25.290230 460 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f97457349ad google::LogMessage::Fail() at ??:0
@ 0x7f974573788a google::LogMessage::SendToLog() at ??:0
@ 0x7f97457344de google::LogMessage::Flush() at ??:0
@ 0x7f9745735a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f975b167512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f975b164976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f975b1ec610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f975b1ec439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9748380345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f974838391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9748383799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f974835eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f974b6386db start_thread at ??:0
@ 0x7f97445ce61f clone at ??:0
W20260925 14:55:25.572302 440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:25.572628 440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:25.572746 440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:25.580248 440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:25.580348 440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:25.580483 440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:55:25.589847 440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:41405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=33279
--tserver_master_addrs=127.17.133.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:25.590986 440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:25.592159 440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:25.604894 480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:25.604894 481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:25.604929 483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:25.605937 440 server_base.cc:1034] running on GCE node
I20260925 14:55:25.606487 440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:25.607342 440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:25.608566 440 hybrid_clock.cc:648] HybridClock initialized: now 1790348125608530 us; error 45 us; skew 500 ppm
I20260925 14:55:25.608949 440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:25 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:25.612092 440 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:55:25.613396 440 webserver.cc:460] Webserver started at http://127.17.133.193:33279/ using document root <none> and password file <none>
I20260925 14:55:25.613993 440 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:25.614090 440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:25.619035 440 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 14:55:25.621790 490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:25.623102 440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:55:25.623268 440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "2cee73d048c4465a94507b827d256797"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "ba1f4bcd96ae8ea60c2ac2493ba6ef73"
server_key_iv: "9b28511edb31e21a536412b265efe175"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.624198 440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:25.636973 440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:25.640563 440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:25.640889 440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:25.641808 440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:25.643448 497 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 14:55:25.648497 440 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:55:25.648651 440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 14:55:25.648780 440 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:55:25.651721 440 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:55:25.651790 440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260925 14:55:25.652123 497 tablet_bootstrap.cc:492] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Bootstrap starting.
I20260925 14:55:25.660558 497 log.cc:824] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:25.675444 497 tablet_bootstrap.cc:492] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:55:25.676169 497 tablet_bootstrap.cc:492] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Bootstrap complete.
I20260925 14:55:25.676708 497 ts_tablet_manager.cc:1397] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.008s
I20260925 14:55:25.686987 497 raft_consensus.cc:348] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:25.688428 497 raft_consensus.cc:729] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cee73d048c4465a94507b827d256797, State: Initialized, Role: FOLLOWER
I20260925 14:55:25.689167 497 consensus_queue.cc:260] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:25.690745 497 ts_tablet_manager.cc:1428] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 14:55:25.695583 440 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:41405
I20260925 14:55:25.695662 605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:41405 every 8 connection(s)
I20260925 14:55:25.696745 440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:25.701301 608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:25.699318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:47323)
0925 14:55:25.700063 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:25.700280 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":313,"threads_started":1}
I20260925 14:55:25.701520 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 440
W20260925 14:55:25.702045 606 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:34843 (0 consecutive failures): Network error: Failed to ping master at 127.17.133.254:34843: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
I20260925 14:55:25.702173 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:34233
--local_ip_for_outbound_sockets=127.17.133.194
--tserver_master_addrs=127.17.133.254:34843
--webserver_port=38207
--webserver_interface=127.17.133.194
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:25.703016 608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:25.702583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:37585)
0925 14:55:25.702767 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:25.702861 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 14:55:25.704041 608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:25.703541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:40385)
0925 14:55:25.703775 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:25.703869 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 14:55:25.704306 606 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:55:26.019440 609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:26.019732 609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:26.019904 609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:26.027338 609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:26.027484 609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:26.027675 609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:55:26.037575 609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:34233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=38207
--tserver_master_addrs=127.17.133.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:26.038931 609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:26.040241 609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:26.053879 615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.054998 617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.061074 614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:26.061968 609 server_base.cc:1034] running on GCE node
I20260925 14:55:26.062866 609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:26.064297 609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:26.065539 609 hybrid_clock.cc:648] HybridClock initialized: now 1790348126065481 us; error 50 us; skew 500 ppm
I20260925 14:55:26.065898 609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:26 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348126, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:26.069058 609 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:55:26.070534 609 webserver.cc:460] Webserver started at http://127.17.133.194:38207/ using document root <none> and password file <none>
I20260925 14:55:26.071261 609 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:26.071420 609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:26.076777 609 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:55:26.081019 624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.082319 609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:55:26.082487 609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "8bf0a857b7a24b13bc3fb4f4011a008a"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-v5vl"
server_key: "016aece6c0dd93f9b8208389bbe39cc5"
server_key_iv: "bbac9f056e2738662d23dc9591f1df20"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.083492 609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:26.097705 609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:26.101406 609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:26.101655 609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:26.102540 609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:26.103739 609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:26.103878 609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.104005 609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:26.104094 609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.137161 609 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:34233
I20260925 14:55:26.137244 736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:34233 every 8 connection(s)
I20260925 14:55:26.138688 609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:26.143182 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.141567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:38565)
0925 14:55:26.142229 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.142427 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":252,"threads_started":1}
W20260925 14:55:26.143831 737 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:34843 (0 consecutive failures): Network error: Failed to ping master at 127.17.133.254:34843: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
I20260925 14:55:26.144682 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.144241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:58461)
0925 14:55:26.144440 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.144535 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 14:55:26.144747 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 609
I20260925 14:55:26.145430 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:46749
--local_ip_for_outbound_sockets=127.17.133.195
--tserver_master_addrs=127.17.133.254:34843
--webserver_port=35661
--webserver_interface=127.17.133.195
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:26.145704 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.145322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:60009)
0925 14:55:26.145473 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.145562 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 14:55:26.145992 737 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:55:26.461656 740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:26.461943 740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:26.462036 740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:26.469012 740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:26.469112 740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:26.469283 740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:55:26.478492 740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:46749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=35661
--tserver_master_addrs=127.17.133.254:34843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:26.479664 740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:26.481024 740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:26.493588 745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.493551 748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.493631 746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:26.494338 740 server_base.cc:1034] running on GCE node
I20260925 14:55:26.494956 740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:26.495832 740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:26.497049 740 hybrid_clock.cc:648] HybridClock initialized: now 1790348126497009 us; error 63 us; skew 500 ppm
I20260925 14:55:26.497324 740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:26 dist-test-slave-v5vl krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348126, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:26.500180 740 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:55:26.501546 740 webserver.cc:460] Webserver started at http://127.17.133.195:35661/ using document root <none> and password file <none>
I20260925 14:55:26.502182 740 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:26.502323 740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:26.507229 740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 14:55:26.509790 755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.511068 740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:26.511301 740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "bb67611044ac45ab8826731ed1522784"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-v5vl"
server_key: "a4a5157e042de1a84367eac05db83165"
server_key_iv: "3cfad6ec043f2a189d9b3908f9e6d1b6"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.512171 740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:26.525925 740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:26.529594 740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:26.529806 740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:26.530637 740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:26.531733 740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:26.531844 740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.531921 740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:26.531975 740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.564318 740 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:46749
I20260925 14:55:26.564406 867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:46749 every 8 connection(s)
I20260925 14:55:26.565850 740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:26.566166 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 740
I20260925 14:55:26.570519 871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.568882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:52425)
0925 14:55:26.569564 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.569761 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":262,"threads_started":1}
W20260925 14:55:26.571085 868 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:34843 (0 consecutive failures): Network error: Failed to ping master at 127.17.133.254:34843: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
I20260925 14:55:26.571774 871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.571513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:59989)
0925 14:55:26.571626 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.571683 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:55:26.572453 871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.572174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:47919)
0925 14:55:26.572293 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.572361 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 14:55:26.572678 868 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:55:26.705523 872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.704864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:60917)
0925 14:55:26.705246 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.705332 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":193,"threads_started":1}
I20260925 14:55:27.147234 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.146438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:33385)
0925 14:55:27.146876 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:27.147009 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
I20260925 14:55:27.293119 874 raft_consensus.cc:482] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:27.293464 874 raft_consensus.cc:504] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:27.295076 874 leader_election.cc:288] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233), bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749)
Sep 25 14:55:27 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
Sep 25 14:55:27 dist-test-slave-v5vl krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:55:27.317332 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.295184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:34233 (local address 127.17.133.193:51621)
0925 14:55:27.295740 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:27.295780 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:27.296389 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:27.300289 (+ 3900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:27.300294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:27.301246 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:27.302038 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.302083 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.303648 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.303653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.305066 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.305078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.306078 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.306085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.306361 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.307178 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:27.307195 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:27.309750 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:27.313436 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:27.313443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:27.313456 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:27.313856 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:27.314328 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:27.314332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:27.314335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:27.314580 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:27.315527 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:27.315534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:27.315808 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 14:55:27.316199 (+ 391us) client_negotiation.cc:240] Negotiation successful
0925 14:55:27.316930 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":323,"spinlock_wait_cycles":17664,"thread_start_us":296,"threads_started":1}
I20260925 14:55:27.317349 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.295950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:51621 (local address 127.17.133.194:34233)
0925 14:55:27.297756 (+ 1806us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:27.297771 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:27.297828 (+ 57us) server_negotiation.cc:407] Connection header received
0925 14:55:27.299667 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:27.299676 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:27.300003 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:27.300332 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:27.302283 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.303465 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.305219 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.305850 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.306059 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.310046 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:27.310053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:27.310067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:27.310100 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:27.313273 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:27.314006 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:27.314015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:27.314022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:27.314224 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:27.314705 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:27.314710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:27.314713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:27.315395 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:27.315588 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:27.316445 (+ 857us) server_negotiation.cc:299] Negotiation successful
0925 14:55:27.316927 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1621,"thread_start_us":1440,"threads_started":1}
I20260925 14:55:27.317368 877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.295686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:46165 (local address 127.17.133.195:46749)
0925 14:55:27.297834 (+ 2148us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:27.297844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:27.297875 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:55:27.299343 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:27.299352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:27.299740 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:27.300088 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:27.302298 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.303459 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.305224 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.305850 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.306069 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.310304 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:27.310310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:27.310323 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:27.310357 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:27.314005 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:27.314742 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:27.314752 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:27.314760 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:27.314979 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:27.315372 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:27.315376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:27.315379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:27.315900 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:27.316099 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:27.316615 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 14:55:27.316982 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2016,"thread_start_us":1908,"threads_started":1}
I20260925 14:55:27.317354 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.295443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:46749 (local address 127.17.133.193:46165)
0925 14:55:27.297289 (+ 1846us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:27.297310 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:27.297430 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:27.300228 (+ 2798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:27.300239 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:27.301268 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:27.302039 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.302064 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.303658 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.303662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.305072 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.305084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.306125 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.306133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.306359 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.307118 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:27.307144 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:27.310049 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:27.314124 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:27.314127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:27.314129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:27.314587 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:27.315099 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:27.315104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:27.315107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:27.315254 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:27.316035 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:27.316041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:27.316209 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 14:55:27.316347 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 14:55:27.316817 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1691,"mutex_wait_us":160,"spinlock_wait_cycles":10496,"thread_start_us":1503,"threads_started":1}
W20260925 14:55:27.321192 492 leader_election.cc:341] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233): Not found: Tablet not found: 5b9961a4de5f43209b83a313a2c0b062
W20260925 14:55:27.321491 491 leader_election.cc:341] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749): Not found: Tablet not found: 5b9961a4de5f43209b83a313a2c0b062
I20260925 14:55:27.321651 491 leader_election.cc:302] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cee73d048c4465a94507b827d256797; no voters: 8bf0a857b7a24b13bc3fb4f4011a008a, bb67611044ac45ab8826731ed1522784
I20260925 14:55:27.322158 874 raft_consensus.cc:2726] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:55:27.573968 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.573151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:43783)
0925 14:55:27.573617 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:27.573754 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":236,"threads_started":1}
I20260925 14:55:27.706780 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.706410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:38323)
0925 14:55:27.706564 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:27.706658 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 14:55:28.148913 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:28.148208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:49315)
0925 14:55:28.148650 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:28.148765 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":204,"threads_started":1}
I20260925 14:55:28.575539 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:28.574911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:50307)
0925 14:55:28.575309 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:28.575389 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":215,"threads_started":1}
I20260925 14:55:28.708262 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:28.707644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:35297)
0925 14:55:28.708027 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:28.708096 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":205,"threads_started":1}
I20260925 14:55:29.150449 883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:29.149786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:41589)
0925 14:55:29.150204 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:29.150283 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":231,"threads_started":1}
I20260925 14:55:29.541579 884 raft_consensus.cc:482] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:29.541785 884 raft_consensus.cc:504] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:29.542411 884 leader_election.cc:288] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233), bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749)
W20260925 14:55:29.543491 491 leader_election.cc:341] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749): Not found: Tablet not found: 5b9961a4de5f43209b83a313a2c0b062
W20260925 14:55:29.543639 492 leader_election.cc:341] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233): Not found: Tablet not found: 5b9961a4de5f43209b83a313a2c0b062
I20260925 14:55:29.543864 492 leader_election.cc:302] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cee73d048c4465a94507b827d256797; no voters: 8bf0a857b7a24b13bc3fb4f4011a008a, bb67611044ac45ab8826731ed1522784
I20260925 14:55:29.544152 884 raft_consensus.cc:2726] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:55:29.577139 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:29.576441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:60323)
0925 14:55:29.576915 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:29.576988 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":288,"threads_started":1}
I20260925 14:55:29.709885 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:29.709109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:36419)
0925 14:55:29.709596 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:29.709704 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":196,"threads_started":1}
I20260925 14:55:30.151944 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:30.151295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:48563)
0925 14:55:30.151698 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:30.151774 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260925 14:55:30.578585 888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:30.577951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:60227)
0925 14:55:30.578312 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:30.578382 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":176,"threads_started":1}
I20260925 14:55:30.711472 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:30.710783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:34047)
0925 14:55:30.711191 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:30.711268 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":198,"threads_started":1}
I20260925 14:55:31.153955 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:31.152857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:55419)
0925 14:55:31.153603 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:31.153705 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":303,"threads_started":1}
I20260925 14:55:31.580004 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:31.579371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:57145)
0925 14:55:31.579780 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:31.579853 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":237,"threads_started":1}
I20260925 14:55:31.712992 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:31.712350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:37997)
0925 14:55:31.712781 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:31.712853 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260925 14:55:32.155673 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.154981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:38149)
0925 14:55:32.155404 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.155490 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":224,"threads_started":1}
I20260925 14:55:32.581497 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.580835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:34291)
0925 14:55:32.581248 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.581323 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
I20260925 14:55:32.714828 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.713942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:58409)
0925 14:55:32.714498 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.714638 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":333,"threads_started":1}
I20260925 14:55:32.809093 896 raft_consensus.cc:482] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:32.809291 896 raft_consensus.cc:504] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:32.809994 896 leader_election.cc:288] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233), bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749)
W20260925 14:55:32.810945 491 leader_election.cc:341] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749): Not found: Tablet not found: 5b9961a4de5f43209b83a313a2c0b062
W20260925 14:55:32.811111 492 leader_election.cc:341] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233): Not found: Tablet not found: 5b9961a4de5f43209b83a313a2c0b062
I20260925 14:55:32.811292 492 leader_election.cc:302] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cee73d048c4465a94507b827d256797; no voters: 8bf0a857b7a24b13bc3fb4f4011a008a, bb67611044ac45ab8826731ed1522784
I20260925 14:55:32.811555 896 raft_consensus.cc:2726] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:55:33.157649 897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.156748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:45527)
0925 14:55:33.157362 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.157456 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":336,"threads_started":1}
I20260925 14:55:33.583006 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.582320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:55833)
0925 14:55:33.582765 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.582840 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":205,"threads_started":1}
I20260925 14:55:33.716310 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.715619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:36993)
0925 14:55:33.716060 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.716139 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260925 14:55:34.159616 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.158755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:35931)
0925 14:55:34.159286 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.159376 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":313,"threads_started":1}
I20260925 14:55:34.584497 901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.583845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:52015)
0925 14:55:34.584242 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.584320 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":214,"threads_started":1}
I20260925 14:55:34.717664 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.717020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:41271)
0925 14:55:34.717434 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.717509 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":209,"threads_started":1}
I20260925 14:55:35.161192 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:35.160463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:56653)
0925 14:55:35.160944 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:35.161025 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":254,"threads_started":1}
I20260925 14:55:35.586406 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:35.585693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:38885)
0925 14:55:35.586175 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:35.586261 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":230,"threads_started":1}
I20260925 14:55:35.719309 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:35.718609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:54137)
0925 14:55:35.719059 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:35.719152 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260925 14:55:36.162981 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.162293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:55355)
0925 14:55:36.162734 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.162817 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260925 14:55:36.547137 907 raft_consensus.cc:482] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:36.547361 907 raft_consensus.cc:504] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf0a857b7a24b13bc3fb4f4011a008a" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34233 } } peers { permanent_uuid: "bb67611044ac45ab8826731ed1522784" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 46749 } } peers { permanent_uuid: "2cee73d048c4465a94507b827d256797" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 41405 } }
I20260925 14:55:36.548072 907 leader_election.cc:288] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233), bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749)
W20260925 14:55:36.548955 492 leader_election.cc:341] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8bf0a857b7a24b13bc3fb4f4011a008a (127.17.133.194:34233): Not found: Tablet not found: 5b9961a4de5f43209b83a313a2c0b062
W20260925 14:55:36.549119 491 leader_election.cc:341] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb67611044ac45ab8826731ed1522784 (127.17.133.195:46749): Not found: Tablet not found: 5b9961a4de5f43209b83a313a2c0b062
I20260925 14:55:36.549279 491 leader_election.cc:302] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cee73d048c4465a94507b827d256797; no voters: 8bf0a857b7a24b13bc3fb4f4011a008a, bb67611044ac45ab8826731ed1522784
I20260925 14:55:36.549558 907 raft_consensus.cc:2726] T 5b9961a4de5f43209b83a313a2c0b062 P 2cee73d048c4465a94507b827d256797 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:55:36.588207 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.587421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:43747)
0925 14:55:36.587934 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.588025 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":266,"threads_started":1}
I20260925 14:55:36.721129 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.720283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:48165)
0925 14:55:36.720842 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.720960 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":242,"threads_started":1}
I20260925 14:55:37.164785 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:37.164014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:55403)
0925 14:55:37.164478 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:37.164563 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
I20260925 14:55:37.589781 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:37.589077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:56571)
0925 14:55:37.589532 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:37.589627 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":236,"threads_started":1}
I20260925 14:55:37.722666 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:37.721987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:46435)
0925 14:55:37.722422 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:37.722503 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":251,"threads_started":1}
I20260925 14:55:38.166349 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.165652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:38943)
0925 14:55:38.166092 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.166171 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260925 14:55:38.591308 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.590640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:37997)
0925 14:55:38.591061 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.591145 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260925 14:55:38.724128 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.723461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:55741)
0925 14:55:38.723881 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.723968 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":230,"threads_started":1}
I20260925 14:55:39.168066 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.167260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:50741)
0925 14:55:39.167759 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:39.167864 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":250,"threads_started":1}
I20260925 14:55:39.593152 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.592336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:54067)
0925 14:55:39.592907 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:39.592996 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":386,"threads_started":1}
I20260925 14:55:39.725734 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.725000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:37939)
0925 14:55:39.725441 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:39.725551 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":210,"threads_started":1}
I20260925 14:55:40.170109 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:40.168973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:39727)
0925 14:55:40.169638 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:40.169806 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":340,"threads_started":1}
I20260925 14:55:40.594676 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:40.593981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:44075)
0925 14:55:40.594410 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:40.594492 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":239,"threads_started":1}
I20260925 14:55:40.727494 921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:40.726840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.193:44047)
0925 14:55:40.727258 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:40.727332 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":224,"threads_started":1}
I20260925 14:55:41.171926 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:41.171161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.194:33963)
0925 14:55:41.171628 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:41.171750 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":219,"threads_started":1}
I20260925 14:55:41.573621 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 440
I20260925 14:55:41.588481 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:41.574607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.17.133.193:41405)
0925 14:55:41.575213 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:41.575231 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:41.575262 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:55:41.575340 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:41.575349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:41.575622 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:41.575803 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:41.577893 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.578867 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.580033 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.580576 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.580812 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:41.583359 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:41.583369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:41.583382 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:41.583410 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:41.586006 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.586626 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.586632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.586634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.586776 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.587083 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.587086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.587088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.587460 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:41.587687 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:41.587953 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:55:41.588134 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":265,"threads_started":1}
I20260925 14:55:41.589732 601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:41.596371 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:41.595637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34843 (local address 127.17.133.195:44721)
0925 14:55:41.596099 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:41.596235 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":262,"threads_started":1}
I20260925 14:55:41.677271 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 440
I20260925 14:55:41.694973 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 609
I20260925 14:55:41.708243 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:41.695734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.17.133.194:34233)
0925 14:55:41.696253 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:41.696257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:41.696271 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:55:41.696322 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:41.696326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:41.696381 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:41.696496 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:41.697807 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.698670 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.699901 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.700317 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.700445 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:41.703314 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:41.703320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:41.703323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:41.703354 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:41.705719 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.706363 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.706370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.706378 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.706515 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.706854 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.706860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.706863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.707117 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:41.707403 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:41.707663 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 14:55:41.707881 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":282,"threads_started":1}
I20260925 14:55:41.708985 732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:41.792570 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 609
I20260925 14:55:41.810822 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 740
I20260925 14:55:41.824031 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:41.811769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.17.133.195:46749)
0925 14:55:41.812351 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:41.812355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:41.812373 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:55:41.812429 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:41.812433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:41.812501 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:41.812671 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:41.813949 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.814733 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.816068 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.816563 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.816710 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:41.819227 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:41.819231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:41.819234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:41.819263 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:41.821913 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.822467 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.822471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.822473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.822617 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.822932 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.822936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.822938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.823189 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:41.823369 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:41.823585 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:55:41.823764 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":321,"threads_started":1}
I20260925 14:55:41.824812 863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:41.908514 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 740
2026-09-25T14:55:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24009 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:42 dist-test-slave-v5vl krb5kdc[932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:42 dist-test-slave-v5vl krb5kdc[932](info): set up 2 sockets
Sep 25 14:55:42 dist-test-slave-v5vl krb5kdc[932](info): commencing operation
krb5kdc: starting...
W20260925 14:55:42.989213 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:55:43 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:55:43.214401 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34641
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:34641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:43.533432 945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:43.533771 945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:43.533938 945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:43.541182 945 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:55:43.541320 945 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:55:43.541407 945 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:55:43.541502 945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:43.541602 945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:43.541716 945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:43.541805 945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:43.541885 945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:43.551438 945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:34641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:43.552754 945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:43.554085 945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:43.565979 950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:43.566051 951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:43.566075 953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:43.567214 945 server_base.cc:1034] running on GCE node
I20260925 14:55:43.567894 945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:43.569267 945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:43.570477 945 hybrid_clock.cc:648] HybridClock initialized: now 1790348143570402 us; error 49 us; skew 500 ppm
I20260925 14:55:43.570761 945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:43 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348143, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:43.573784 945 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:55:43.575311 945 webserver.cc:460] Webserver started at http://127.17.133.254:38635/ using document root <none> and password file <none>
I20260925 14:55:43.575996 945 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:43.576079 945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:43.576323 945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:43.579152 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "d6040f5d1d2a46c399e645c2c6e1226c"
format_stamp: "Formatted at 2026-09-25 14:55:43 on dist-test-slave-v5vl"
server_key: "105bbf78dca52c210c8b1d427da39f32"
server_key_iv: "0182284bdaeb18addb2b44e543c47437"
server_key_version: "encryptionkey@0"
I20260925 14:55:43.579865 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "d6040f5d1d2a46c399e645c2c6e1226c"
format_stamp: "Formatted at 2026-09-25 14:55:43 on dist-test-slave-v5vl"
server_key: "105bbf78dca52c210c8b1d427da39f32"
server_key_iv: "0182284bdaeb18addb2b44e543c47437"
server_key_version: "encryptionkey@0"
I20260925 14:55:43.584766 945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 14:55:43.587878 960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:43.589191 945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:43.589365 945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "d6040f5d1d2a46c399e645c2c6e1226c"
format_stamp: "Formatted at 2026-09-25 14:55:43 on dist-test-slave-v5vl"
server_key: "105bbf78dca52c210c8b1d427da39f32"
server_key_iv: "0182284bdaeb18addb2b44e543c47437"
server_key_version: "encryptionkey@0"
I20260925 14:55:43.589573 945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:43.601581 945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:43.605366 945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:43.605618 945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:43.623471 945 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:34641
I20260925 14:55:43.623474 1011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:34641 every 8 connection(s)
I20260925 14:55:43.624789 945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:43.625561 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 945
I20260925 14:55:43.625823 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:55:43.626587 17943 external_mini_cluster.cc:1428] Setting key 3a719552f68f060b26a137685789b518
I20260925 14:55:43.629101 1012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 14:55:43 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:43.641801 1012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Bootstrap starting.
I20260925 14:55:43.645799 1012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:43.646904 1012 log.cc:824] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:43.649878 1012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: No bootstrap required, opened a new log
I20260925 14:55:43.649909 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:43.628464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.17.133.254:34641)
0925 14:55:43.629326 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:43.629336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:43.629380 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:55:43.630955 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:43.630973 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:43.631328 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:43.631631 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:43.633450 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.634776 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.636645 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.637227 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.637364 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:43.641994 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:43.642004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:43.642021 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:43.642077 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:43.645789 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:43.646481 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:43.646488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:43.646494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:43.646683 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:43.647015 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:43.647019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:43.647022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:43.647642 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:43.647850 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:43.648339 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 14:55:43.648717 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":348,"threads_started":1}
I20260925 14:55:43.657732 1012 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:55:43.658041 1012 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:43.658103 1012 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6040f5d1d2a46c399e645c2c6e1226c, State: Initialized, Role: FOLLOWER
I20260925 14:55:43.658602 1012 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:55:43.658855 1012 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:43.658955 1012 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:43.659075 1012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:43.661129 1012 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:55:43.661706 1012 leader_election.cc:302] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6040f5d1d2a46c399e645c2c6e1226c; no voters:
I20260925 14:55:43.662272 1012 leader_election.cc:288] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:43.662397 1017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:43.663074 1017 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 1 LEADER]: Becoming Leader. State: Replica: d6040f5d1d2a46c399e645c2c6e1226c, State: Running, Role: LEADER
I20260925 14:55:43.663559 1017 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:55:43.664167 1012 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:43.666302 1018 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } } }
I20260925 14:55:43.666252 1019 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6040f5d1d2a46c399e645c2c6e1226c. Latest consensus state: current_term: 1 leader_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } } }
I20260925 14:55:43.666721 1018 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:43.666741 1019 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:43.669219 1024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:43.675395 1024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:43.685689 1024 catalog_manager.cc:1347] Generated new cluster ID: e2e2adb6bca74e5695fcbbba895f218d
I20260925 14:55:43.685796 1024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:43.708370 1024 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:43.710311 1024 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:43.717311 1024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Generated new TSK 0
I20260925 14:55:43.718138 1024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:55:43.792130 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34641
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:44.127444 1038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:44.127705 1038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:44.127882 1038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:44.135172 1038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:44.135279 1038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:44.135447 1038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:55:44.145026 1038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:44.146279 1038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:44.147588 1038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:44.160852 1046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:44.160895 1043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:44.160811 1044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:44.161890 1038 server_base.cc:1034] running on GCE node
I20260925 14:55:44.162488 1038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:44.163424 1038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:44.164726 1038 hybrid_clock.cc:648] HybridClock initialized: now 1790348144164673 us; error 53 us; skew 500 ppm
I20260925 14:55:44.165071 1038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:44 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348144, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:44.168216 1038 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:55:44.172192 1038 webserver.cc:460] Webserver started at http://127.17.133.193:38097/ using document root <none> and password file <none>
I20260925 14:55:44.173159 1038 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:44.173305 1038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:44.173614 1038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:44.177095 1038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "d7588880f61a4cd5bf8b626a87356f92"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "0fb961a42437692b03472284c0fcc18f"
server_key_iv: "d04100795e185bf9c3fe259a63ed1683"
server_key_version: "encryptionkey@0"
I20260925 14:55:44.177865 1038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "d7588880f61a4cd5bf8b626a87356f92"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "0fb961a42437692b03472284c0fcc18f"
server_key_iv: "d04100795e185bf9c3fe259a63ed1683"
server_key_version: "encryptionkey@0"
I20260925 14:55:44.182950 1038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:55:44.186372 1053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:44.187631 1038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:55:44.187801 1038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "d7588880f61a4cd5bf8b626a87356f92"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "0fb961a42437692b03472284c0fcc18f"
server_key_iv: "d04100795e185bf9c3fe259a63ed1683"
server_key_version: "encryptionkey@0"
I20260925 14:55:44.187990 1038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:44.207950 1038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:44.212054 1038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:44.212373 1038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:44.213378 1038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:44.214717 1038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:44.214790 1038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:44.214947 1038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:44.215014 1038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:44.248243 1038 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:38953
I20260925 14:55:44.248297 1165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:38953 every 8 connection(s)
I20260925 14:55:44.249557 1038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:44.250419 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 1038
I20260925 14:55:44.250626 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:44.251433 17943 external_mini_cluster.cc:1428] Setting key 25934b8e0e1d4301296d08aeead6eba5
Sep 25 14:55:44 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348144, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:44.271701 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:44.252372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:34737 (local address 127.17.133.254:34641)
0925 14:55:44.252830 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:44.252835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:44.253965 (+ 1130us) server_negotiation.cc:407] Connection header received
0925 14:55:44.255957 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:44.255966 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:44.256065 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:44.256169 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:44.257741 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.258903 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.260420 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.260928 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.261025 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.264815 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:44.264824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:44.264832 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:44.264878 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:44.268447 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:44.269220 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:44.269229 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:44.269235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:44.269390 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:44.269780 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:44.269787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:44.269793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:44.270159 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:44.270316 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:44.270958 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 14:55:44.271209 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":239,"threads_started":1}
I20260925 14:55:44.272205 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:44.252999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:34737)
0925 14:55:44.253797 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:44.253844 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:44.255537 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:44.256385 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:44.256393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:44.257033 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:44.257576 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.257594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.259055 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.259059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.260280 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.260291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.261157 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.261166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.261498 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.262258 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:44.262284 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:44.264562 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:44.268625 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:44.268632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:44.268667 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:44.269054 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:44.269494 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:44.269497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:44.269499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:44.269665 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:44.270283 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:44.270288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:44.270566 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 14:55:44.270769 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 14:55:44.271131 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":331,"threads_started":1}
I20260925 14:55:44.273995 1166 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34641
I20260925 14:55:44.274300 1166 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:44.274984 1166 heartbeater.cc:511] Master 127.17.133.254:34641 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
I20260925 14:55:44.277426 977 ts_manager.cc:194] Registered new tserver with Master: d7588880f61a4cd5bf8b626a87356f92 (127.17.133.193:38953)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
I20260925 14:55:44.279136 977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:34737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:55:44.306078 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34641
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:44.624104 1172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:44.624351 1172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:44.624505 1172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:44.631283 1172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:44.631376 1172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:44.631553 1172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:55:44.640820 1172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:44.641934 1172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:44.643146 1172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:44.655567 1177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:44.655541 1178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:44.655606 1180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:44.656342 1172 server_base.cc:1034] running on GCE node
I20260925 14:55:44.656919 1172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:44.657676 1172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:44.658927 1172 hybrid_clock.cc:648] HybridClock initialized: now 1790348144658880 us; error 34 us; skew 500 ppm
I20260925 14:55:44.659219 1172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:44 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348144, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:44.662181 1172 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:55:44.663398 1172 webserver.cc:460] Webserver started at http://127.17.133.194:33943/ using document root <none> and password file <none>
I20260925 14:55:44.664001 1172 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:44.664160 1172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:44.664481 1172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:44.667320 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "c30625681fe747e0b00297efd5716c57"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "b0b2e3ebd03abe46e978084dfe7e4bca"
server_key_iv: "f517477017e6a15510bd0c4e847ce641"
server_key_version: "encryptionkey@0"
I20260925 14:55:44.668071 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "c30625681fe747e0b00297efd5716c57"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "b0b2e3ebd03abe46e978084dfe7e4bca"
server_key_iv: "f517477017e6a15510bd0c4e847ce641"
server_key_version: "encryptionkey@0"
I20260925 14:55:44.673131 1172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 14:55:44.676702 1187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:44.677909 1172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:55:44.678030 1172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "c30625681fe747e0b00297efd5716c57"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "b0b2e3ebd03abe46e978084dfe7e4bca"
server_key_iv: "f517477017e6a15510bd0c4e847ce641"
server_key_version: "encryptionkey@0"
I20260925 14:55:44.678179 1172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:44.709080 1172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:44.712949 1172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:44.713177 1172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:44.714043 1172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:44.715255 1172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:44.715325 1172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:44.715413 1172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:44.715456 1172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:44.747295 1172 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:43323
I20260925 14:55:44.747335 1299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:43323 every 8 connection(s)
I20260925 14:55:44.748544 1172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:44.757714 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 1172
I20260925 14:55:44.757899 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:44.758348 17943 external_mini_cluster.cc:1428] Setting key 9a98c9c1fa10946cc3522267d45461e0
Sep 25 14:55:44 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348144, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:44.769457 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:44.750945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:36007 (local address 127.17.133.254:34641)
0925 14:55:44.751146 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:44.751150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:44.752226 (+ 1076us) server_negotiation.cc:407] Connection header received
0925 14:55:44.754022 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:44.754027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:44.754148 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:44.754259 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:44.755857 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.756664 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.758271 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.758786 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.758920 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.762285 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:44.762298 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:44.762306 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:44.762351 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:44.766138 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:44.766864 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:44.766873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:44.766879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:44.767035 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:44.767494 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:44.767501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:44.767506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:44.767895 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:44.768044 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:44.768738 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 14:55:44.769002 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 14:55:44.770290 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:44.751420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:36007)
0925 14:55:44.752055 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:44.752097 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:44.753753 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:44.754491 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:44.754498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:44.754983 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:44.755598 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.755622 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.756811 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.756815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.758135 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.758146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.759009 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.759016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.759238 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.759890 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:44.759915 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:44.762009 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:44.766286 (+ 4277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:44.766293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:44.766307 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:44.766743 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:44.767175 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:44.767178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:44.767181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:44.767381 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:44.768012 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:44.768017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:44.768349 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 14:55:44.768577 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 14:55:44.769040 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":253,"threads_started":1}
I20260925 14:55:44.771852 1300 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34641
I20260925 14:55:44.772202 1300 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:44.773064 1300 heartbeater.cc:511] Master 127.17.133.254:34641 requested a full tablet report, sending...
I20260925 14:55:44.774645 977 ts_manager.cc:194] Registered new tserver with Master: c30625681fe747e0b00297efd5716c57 (127.17.133.194:43323)
I20260925 14:55:44.775893 977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:36007
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:55:44.812012 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34641
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:45.137077 1305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:45.137327 1305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:45.137418 1305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:45.144500 1305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:45.144668 1305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:45.144807 1305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:55:45.154182 1305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:55:45.155325 1305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:45.156504 1305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:45.169431 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:45.169406 1313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:45.169469 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:45.170603 1305 server_base.cc:1034] running on GCE node
I20260925 14:55:45.171240 1305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:45.172194 1305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:45.173414 1305 hybrid_clock.cc:648] HybridClock initialized: now 1790348145173325 us; error 57 us; skew 500 ppm
I20260925 14:55:45.173729 1305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:55:45 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348145, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:45.176755 1305 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:55:45.178063 1305 webserver.cc:460] Webserver started at http://127.17.133.195:42373/ using document root <none> and password file <none>
I20260925 14:55:45.178660 1305 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:45.178791 1305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:45.179095 1305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:45.181994 1305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "a37d21a350aa4bd8837e75f19d6ea917"
format_stamp: "Formatted at 2026-09-25 14:55:45 on dist-test-slave-v5vl"
server_key: "a0606ff8447def512c36b0cb6aa435bd"
server_key_iv: "02f9cedccde45e13fcd1c50bad2c4fa8"
server_key_version: "encryptionkey@0"
I20260925 14:55:45.182770 1305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "a37d21a350aa4bd8837e75f19d6ea917"
format_stamp: "Formatted at 2026-09-25 14:55:45 on dist-test-slave-v5vl"
server_key: "a0606ff8447def512c36b0cb6aa435bd"
server_key_iv: "02f9cedccde45e13fcd1c50bad2c4fa8"
server_key_version: "encryptionkey@0"
I20260925 14:55:45.187413 1305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 14:55:45.190986 1320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:45.192231 1305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:55:45.192452 1305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "a37d21a350aa4bd8837e75f19d6ea917"
format_stamp: "Formatted at 2026-09-25 14:55:45 on dist-test-slave-v5vl"
server_key: "a0606ff8447def512c36b0cb6aa435bd"
server_key_iv: "02f9cedccde45e13fcd1c50bad2c4fa8"
server_key_version: "encryptionkey@0"
I20260925 14:55:45.192673 1305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:45.210044 1305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:45.213866 1305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:45.214143 1305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:45.215071 1305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:45.216320 1305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:45.216431 1305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:45.216584 1305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:45.216701 1305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:45.249341 1305 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:45953
I20260925 14:55:45.249364 1432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:45953 every 8 connection(s)
I20260925 14:55:45.250725 1305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:45.253906 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 1305
I20260925 14:55:45.254101 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:45.254556 17943 external_mini_cluster.cc:1428] Setting key 8a4a45d26e57c57b061c9ae1408e1f97
Sep 25 14:55:45 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348145, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:55:45.270021 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.253646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:59211 (local address 127.17.133.254:34641)
0925 14:55:45.253883 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.253892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.255077 (+ 1185us) server_negotiation.cc:407] Connection header received
0925 14:55:45.256965 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.256971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.257084 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.257189 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:45.258634 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.259388 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.260877 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.261273 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.261370 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.264490 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:45.264494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:45.264497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:45.264528 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:45.266986 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.267586 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.267589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.267591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.267728 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.268167 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.268171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.268173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.268463 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:45.268684 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.269566 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.269787 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 14:55:45.270913 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.254073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:59211)
0925 14:55:45.254914 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:45.254964 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:45.256717 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:45.257441 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:45.257449 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:45.257960 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:45.258490 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.258506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.259504 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.259508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.260756 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.260767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.261452 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.261457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.261637 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.262231 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:45.262256 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:45.264324 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:45.267110 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:45.267116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:45.267127 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:45.267495 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:45.267854 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:45.267859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:45.267863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:45.268034 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:45.268706 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:45.268712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:45.269168 (+ 456us) client_negotiation.cc:769] Sending connection context
0925 14:55:45.269431 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 14:55:45.269932 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":324,"threads_started":1}
I20260925 14:55:45.272353 1433 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34641
I20260925 14:55:45.272688 1433 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:45.273453 1433 heartbeater.cc:511] Master 127.17.133.254:34641 requested a full tablet report, sending...
I20260925 14:55:45.274829 977 ts_manager.cc:194] Registered new tserver with Master: a37d21a350aa4bd8837e75f19d6ea917 (127.17.133.195:45953)
I20260925 14:55:45.275601 977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:59211
I20260925 14:55:45.281889 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:45.282157 1166 heartbeater.cc:503] Master 127.17.133.254:34641 was elected leader, sending a full tablet report...
I20260925 14:55:45.298460 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.286809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.17.133.254:34641)
0925 14:55:45.286964 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.286969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.287322 (+ 353us) server_negotiation.cc:407] Connection header received
0925 14:55:45.287432 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.287435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.287489 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.287565 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:45.289009 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.289691 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.290945 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.291358 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.291442 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.293949 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:45.293953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:45.293956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:45.293983 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:45.296326 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.296972 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.296976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.296978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.297101 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.297408 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.297411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.297413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.297695 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:45.297873 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.298101 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.298242 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:55:45.301939 977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:45.308632 977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:45.329312 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.319966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.17.133.193:38953)
0925 14:55:45.321415 (+ 1449us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.321426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.321456 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:55:45.321558 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.321566 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.321819 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.322808 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:45.323068 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.324655 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.326993 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.328157 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.328364 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.328529 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.328814 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.328967 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":1205,"threads_started":1}
I20260925 14:55:45.329469 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.319645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:38953 (local address 127.0.0.1:56284)
0925 14:55:45.320609 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:45.320653 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:45.320821 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:45.322048 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:45.322054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:45.322090 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:45.322566 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.322581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.324785 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.324790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.326792 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.326808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.328361 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.328367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.328537 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.328618 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 14:55:45.328764 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 14:55:45.328958 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":181,"thread_start_us":568,"threads_started":1}
I20260925 14:55:45.330336 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.319880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:43323 (local address 127.0.0.1:59946)
0925 14:55:45.321254 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:45.321286 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:45.321383 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:45.322267 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:45.322271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:45.322302 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:45.322764 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.322774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.324948 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.324954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.327372 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.327388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.329265 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.329280 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.329559 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.329690 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 14:55:45.329780 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:55:45.329911 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"mutex_wait_us":124,"thread_start_us":1120,"threads_started":1}
I20260925 14:55:45.330497 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.321830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.17.133.195:45953)
0925 14:55:45.322615 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.322621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.322646 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:55:45.322714 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.322718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.322855 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.323046 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:45.323624 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.324970 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.327577 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.328993 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.329754 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.329916 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.329988 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.330138 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":517,"threads_started":1}
I20260925 14:55:45.330363 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.321342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:45953 (local address 127.0.0.1:49624)
0925 14:55:45.321813 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:45.321834 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:45.321935 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:45.323026 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:45.323031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:45.323055 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:45.323432 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.323446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.325118 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.325125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.327413 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.327424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.329301 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.329307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.329563 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.329734 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 14:55:45.329827 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:55:45.329958 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":141,"thread_start_us":686,"threads_started":1}
I20260925 14:55:45.331301 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.320370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.17.133.194:43323)
0925 14:55:45.321711 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.321719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.321743 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:55:45.321827 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.321834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.322060 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.322948 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:45.323369 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.324814 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.327568 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.329002 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.330450 (+ 1448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.330599 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.330663 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.331025 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":1602,"threads_started":1}
I20260925 14:55:45.333648 1101 tablet_service.cc:1467] Processing CreateTablet for tablet 11fa60693a26437aae3976756b96cf82 (DEFAULT_TABLE table=test-table [id=87a8f524fc0e472eab873ad491dd7360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:45.334179 1368 tablet_service.cc:1467] Processing CreateTablet for tablet 11fa60693a26437aae3976756b96cf82 (DEFAULT_TABLE table=test-table [id=87a8f524fc0e472eab873ad491dd7360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:45.334882 1101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fa60693a26437aae3976756b96cf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:45.334558 1235 tablet_service.cc:1467] Processing CreateTablet for tablet 11fa60693a26437aae3976756b96cf82 (DEFAULT_TABLE table=test-table [id=87a8f524fc0e472eab873ad491dd7360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:45.335525 1368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fa60693a26437aae3976756b96cf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:45.335973 1235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fa60693a26437aae3976756b96cf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:45.345445 1451 tablet_bootstrap.cc:492] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: Bootstrap starting.
I20260925 14:55:45.348680 1451 tablet_bootstrap.cc:654] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:45.350070 1451 log.cc:824] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:45.350186 1452 tablet_bootstrap.cc:492] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: Bootstrap starting.
I20260925 14:55:45.350981 1453 tablet_bootstrap.cc:492] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: Bootstrap starting.
I20260925 14:55:45.353331 1451 tablet_bootstrap.cc:492] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: No bootstrap required, opened a new log
I20260925 14:55:45.353598 1451 ts_tablet_manager.cc:1397] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260925 14:55:45.354633 1452 tablet_bootstrap.cc:654] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:45.354705 1453 tablet_bootstrap.cc:654] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:45.355851 1452 log.cc:824] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:45.355851 1453 log.cc:824] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:45.358827 1453 tablet_bootstrap.cc:492] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: No bootstrap required, opened a new log
I20260925 14:55:45.359090 1453 ts_tablet_manager.cc:1397] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260925 14:55:45.359670 1452 tablet_bootstrap.cc:492] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: No bootstrap required, opened a new log
I20260925 14:55:45.359913 1452 ts_tablet_manager.cc:1397] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260925 14:55:45.363139 1451 raft_consensus.cc:348] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } }
I20260925 14:55:45.363618 1451 raft_consensus.cc:374] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:45.363749 1451 raft_consensus.cc:729] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7588880f61a4cd5bf8b626a87356f92, State: Initialized, Role: FOLLOWER
I20260925 14:55:45.364480 1451 consensus_queue.cc:260] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } }
I20260925 14:55:45.366264 1451 ts_tablet_manager.cc:1428] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260925 14:55:45.368680 1452 raft_consensus.cc:348] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } }
I20260925 14:55:45.369161 1452 raft_consensus.cc:374] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:45.369282 1452 raft_consensus.cc:729] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30625681fe747e0b00297efd5716c57, State: Initialized, Role: FOLLOWER
I20260925 14:55:45.369463 1453 raft_consensus.cc:348] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } }
I20260925 14:55:45.369891 1453 raft_consensus.cc:374] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:45.370013 1453 raft_consensus.cc:729] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37d21a350aa4bd8837e75f19d6ea917, State: Initialized, Role: FOLLOWER
I20260925 14:55:45.369971 1452 consensus_queue.cc:260] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } }
I20260925 14:55:45.370585 1453 consensus_queue.cc:260] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } }
I20260925 14:55:45.372287 1433 heartbeater.cc:503] Master 127.17.133.254:34641 was elected leader, sending a full tablet report...
I20260925 14:55:45.372359 1453 ts_tablet_manager.cc:1428] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 14:55:45.373256 1300 heartbeater.cc:503] Master 127.17.133.254:34641 was elected leader, sending a full tablet report...
I20260925 14:55:45.373955 1452 ts_tablet_manager.cc:1428] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260925 14:55:45.441902 1457 raft_consensus.cc:482] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:45.442209 1457 raft_consensus.cc:504] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } }
I20260925 14:55:45.443797 1457 leader_election.cc:288] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c30625681fe747e0b00297efd5716c57 (127.17.133.194:43323), a37d21a350aa4bd8837e75f19d6ea917 (127.17.133.195:45953)
I20260925 14:55:45.450259 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.444318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:43215 (local address 127.17.133.194:43323)
0925 14:55:45.444494 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.444499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.445082 (+ 583us) server_negotiation.cc:407] Connection header received
0925 14:55:45.445298 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.445303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.445395 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.445514 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:45.446069 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.446758 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.448616 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.449384 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.449566 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.449641 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.449892 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.450011 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:55:45.450328 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.444262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:43323 (local address 127.17.133.193:43215)
0925 14:55:45.444982 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:45.445003 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:45.445134 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:45.445538 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:45.445542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:45.445580 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:45.445924 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.445934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.446896 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.446904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.448448 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.448459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.449590 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.449596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.449738 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.449761 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:55:45.449858 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:55:45.449970 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":429,"threads_started":1}
I20260925 14:55:45.450855 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.445280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:57459 (local address 127.17.133.195:45953)
0925 14:55:45.445501 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.445506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.445828 (+ 322us) server_negotiation.cc:407] Connection header received
0925 14:55:45.446045 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.446050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.446116 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.446215 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:45.446735 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.447477 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.449159 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.449901 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.450079 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.450149 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.450481 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.450593 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:55:45.451370 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.445099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:45953 (local address 127.17.133.193:57459)
0925 14:55:45.445720 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:45.445741 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:45.445894 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:45.446238 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:45.446241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:45.446265 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:45.446578 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.446585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.447620 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.447625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.449011 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.449023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.450120 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.450127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.450271 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.450293 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:55:45.451022 (+ 729us) client_negotiation.cc:240] Negotiation successful
0925 14:55:45.451156 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":407,"threads_started":1}
I20260925 14:55:45.451578 1255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fa60693a26437aae3976756b96cf82" candidate_uuid: "d7588880f61a4cd5bf8b626a87356f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30625681fe747e0b00297efd5716c57" is_pre_election: true
I20260925 14:55:45.452067 1255 raft_consensus.cc:2454] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7588880f61a4cd5bf8b626a87356f92 in term 0.
I20260925 14:55:45.452178 1388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fa60693a26437aae3976756b96cf82" candidate_uuid: "d7588880f61a4cd5bf8b626a87356f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37d21a350aa4bd8837e75f19d6ea917" is_pre_election: true
I20260925 14:55:45.452718 1388 raft_consensus.cc:2454] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7588880f61a4cd5bf8b626a87356f92 in term 0.
I20260925 14:55:45.452800 1057 leader_election.cc:302] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30625681fe747e0b00297efd5716c57, d7588880f61a4cd5bf8b626a87356f92; no voters:
I20260925 14:55:45.453217 1457 raft_consensus.cc:2781] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:45.453368 1457 raft_consensus.cc:482] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:45.453487 1457 raft_consensus.cc:3037] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:45.455574 1457 raft_consensus.cc:504] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } }
I20260925 14:55:45.456111 1457 leader_election.cc:288] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [CANDIDATE]: Term 1 election: Requested vote from peers c30625681fe747e0b00297efd5716c57 (127.17.133.194:43323), a37d21a350aa4bd8837e75f19d6ea917 (127.17.133.195:45953)
I20260925 14:55:45.456712 1255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fa60693a26437aae3976756b96cf82" candidate_uuid: "d7588880f61a4cd5bf8b626a87356f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30625681fe747e0b00297efd5716c57"
I20260925 14:55:45.456723 1388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fa60693a26437aae3976756b96cf82" candidate_uuid: "d7588880f61a4cd5bf8b626a87356f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37d21a350aa4bd8837e75f19d6ea917"
I20260925 14:55:45.456905 1388 raft_consensus.cc:3037] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:45.456897 1255 raft_consensus.cc:3037] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:45.458989 1388 raft_consensus.cc:2454] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7588880f61a4cd5bf8b626a87356f92 in term 1.
I20260925 14:55:45.459169 1255 raft_consensus.cc:2454] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7588880f61a4cd5bf8b626a87356f92 in term 1.
I20260925 14:55:45.459327 1057 leader_election.cc:302] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a37d21a350aa4bd8837e75f19d6ea917, d7588880f61a4cd5bf8b626a87356f92; no voters:
I20260925 14:55:45.459594 1457 raft_consensus.cc:2781] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:45.460037 1457 raft_consensus.cc:686] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 1 LEADER]: Becoming Leader. State: Replica: d7588880f61a4cd5bf8b626a87356f92, State: Running, Role: LEADER
I20260925 14:55:45.460487 1457 consensus_queue.cc:237] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } }
I20260925 14:55:45.465121 977 catalog_manager.cc:5495] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7588880f61a4cd5bf8b626a87356f92 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "d7588880f61a4cd5bf8b626a87356f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:45.482295 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.477344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.17.133.193:38953)
0925 14:55:45.477515 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.477519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.477631 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:55:45.477808 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.477812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.477943 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.478061 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:45.478389 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.479066 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.480329 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.480803 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.480935 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.481334 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:45.481851 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:45.481982 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.482031 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.482091 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:55:45.490018 1388 raft_consensus.cc:1261] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [term 1 FOLLOWER]: Refusing update from remote peer d7588880f61a4cd5bf8b626a87356f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:45.490069 1255 raft_consensus.cc:1261] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [term 1 FOLLOWER]: Refusing update from remote peer d7588880f61a4cd5bf8b626a87356f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:45.490948 1462 consensus_queue.cc:1035] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:45.491328 1457 consensus_queue.cc:1035] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:55:45.500571 1301 tablet.cc:2369] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:45.501914 1167 tablet.cc:2369] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:45.502144 1465 mvcc.cc:205] Tried to move back new op lower bound from 7333266003917905920 to 7333266003809869824. Current Snapshot: MvccSnapshot[applied={T|T < 7333266003917905920}]
W20260925 14:55:45.502418 1434 tablet.cc:2369] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:45.518869 977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:45.519148 977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:45.522701 977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Sending DeleteTablet for 3 replicas of tablet 11fa60693a26437aae3976756b96cf82
I20260925 14:55:45.523808 1101 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fa60693a26437aae3976756b96cf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:45 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:56284
I20260925 14:55:45.523916 1368 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fa60693a26437aae3976756b96cf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:45 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:49624
I20260925 14:55:45.523998 1235 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fa60693a26437aae3976756b96cf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:45 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:59946
I20260925 14:55:45.524099 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 1038
I20260925 14:55:45.536316 1475 tablet_replica.cc:330] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: stopping tablet replica
I20260925 14:55:45.536810 1475 raft_consensus.cc:2229] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:45.537639 1475 raft_consensus.cc:2258] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:45.539762 1475 ts_tablet_manager.cc:1905] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:45.541460 1476 tablet_replica.cc:330] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: stopping tablet replica
I20260925 14:55:45.541605 1477 tablet_replica.cc:330] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: stopping tablet replica
I20260925 14:55:45.541963 1476 raft_consensus.cc:2229] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:45.542026 1477 raft_consensus.cc:2229] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:45.542495 1476 raft_consensus.cc:2258] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:45.542543 1477 raft_consensus.cc:2258] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:45.544514 1477 ts_tablet_manager.cc:1905] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:45.547206 1476 ts_tablet_manager.cc:1905] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:45.548013 1475 ts_tablet_manager.cc:1918] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:45.548264 1475 log.cc:1196] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/11fa60693a26437aae3976756b96cf82
I20260925 14:55:45.548904 1475 ts_tablet_manager.cc:1939] T 11fa60693a26437aae3976756b96cf82 P d7588880f61a4cd5bf8b626a87356f92: Deleting consensus metadata
I20260925 14:55:45.550740 962 catalog_manager.cc:4841] TS d7588880f61a4cd5bf8b626a87356f92 (127.17.133.193:38953): tablet 11fa60693a26437aae3976756b96cf82 (table test-table [id=87a8f524fc0e472eab873ad491dd7360]) successfully deleted
I20260925 14:55:45.551509 1477 ts_tablet_manager.cc:1918] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:45.551687 1477 log.cc:1196] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/11fa60693a26437aae3976756b96cf82
I20260925 14:55:45.552205 1477 ts_tablet_manager.cc:1939] T 11fa60693a26437aae3976756b96cf82 P a37d21a350aa4bd8837e75f19d6ea917: Deleting consensus metadata
I20260925 14:55:45.553856 964 catalog_manager.cc:4841] TS a37d21a350aa4bd8837e75f19d6ea917 (127.17.133.195:45953): tablet 11fa60693a26437aae3976756b96cf82 (table test-table [id=87a8f524fc0e472eab873ad491dd7360]) successfully deleted
I20260925 14:55:45.553916 1476 ts_tablet_manager.cc:1918] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:45.554034 1476 log.cc:1196] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/11fa60693a26437aae3976756b96cf82
I20260925 14:55:45.554582 1476 ts_tablet_manager.cc:1939] T 11fa60693a26437aae3976756b96cf82 P c30625681fe747e0b00297efd5716c57: Deleting consensus metadata
Sep 25 14:55:45 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:55:45.556175 964 catalog_manager.cc:4841] TS c30625681fe747e0b00297efd5716c57 (127.17.133.194:43323): tablet 11fa60693a26437aae3976756b96cf82 (table test-table [id=87a8f524fc0e472eab873ad491dd7360]) successfully deleted
I20260925 14:55:45.563110 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.524910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.17.133.193:38953)
0925 14:55:45.525092 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.525100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.536680 (+ 11580us) server_negotiation.cc:407] Connection header received
0925 14:55:45.537015 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.537023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.537169 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.537256 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:45.539907 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.541939 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.544035 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.544858 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.544953 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.556111 (+ 11158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:45.556125 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:45.556141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:45.556186 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:45.560175 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.560930 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.560941 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.560945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.561143 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.561506 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.561511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.561514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.561931 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:45.562225 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.562445 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.562695 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":54}
I20260925 14:55:45.563777 1161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:45.658545 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 1038
I20260925 14:55:45.676431 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 1172
Sep 25 14:55:45 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:55:45.690526 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.677348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.17.133.194:43323)
0925 14:55:45.677526 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.677530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.677550 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:55:45.677604 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.677608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.677673 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.677757 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:45.679038 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.679780 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.681058 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.681451 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.681608 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.685849 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:45.685859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:45.685873 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:45.685902 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:45.688232 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.688892 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.688899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.688901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.689017 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.689316 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.689319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.689320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.689635 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:45.689836 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.690126 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.690286 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:55:45.691234 1295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:45.779101 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 1172
I20260925 14:55:45.796020 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 1305
Sep 25 14:55:45 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:55:45.810858 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.796793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.17.133.195:45953)
0925 14:55:45.796979 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.796985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.797010 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:55:45.797170 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.797174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.797236 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.797328 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:45.798614 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.799345 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.800766 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.801140 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.801228 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.805552 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:45.805565 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:45.805581 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:45.805614 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:45.808294 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.809132 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.809139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.809141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.809277 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:45.809598 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:45.809601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:45.809603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:45.809919 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:45.810126 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:45.810360 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 14:55:45.810512 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:55:45.811510 1428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:45.899101 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 1305
I20260925 14:55:45.915439 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 945
I20260925 14:55:45.916692 1007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:55:46.025416 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 945
I20260925 14:56:06.046555 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34641
--webserver_interface=127.17.133.254
--webserver_port=38635
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:34641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:06.370860 1483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:06.371143 1483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:06.371325 1483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:06.378492 1483 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:56:06.378592 1483 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:56:06.378715 1483 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:56:06.378810 1483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:06.378902 1483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:06.379016 1483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:06.379088 1483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:06.379165 1483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:06.389199 1483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:34641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:06.390453 1483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:06.391806 1483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:06.413682 1490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:06.413758 1489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:06.413734 1492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:06.414072 1483 server_base.cc:1034] running on GCE node
I20260925 14:56:06.415202 1483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:06.416785 1483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:06.418035 1483 hybrid_clock.cc:648] HybridClock initialized: now 1790348166417965 us; error 58 us; skew 500 ppm
I20260925 14:56:06.418433 1483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:06 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348166, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:06.431663 1483 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:56:06.433321 1483 webserver.cc:460] Webserver started at http://127.17.133.254:38635/ using document root <none> and password file <none>
I20260925 14:56:06.434082 1483 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:06.434180 1483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:06.440100 1483 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 14:56:06.443091 1499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:06.444438 1483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:56:06.444659 1483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "d6040f5d1d2a46c399e645c2c6e1226c"
format_stamp: "Formatted at 2026-09-25 14:55:43 on dist-test-slave-v5vl"
server_key: "105bbf78dca52c210c8b1d427da39f32"
server_key_iv: "0182284bdaeb18addb2b44e543c47437"
server_key_version: "encryptionkey@0"
I20260925 14:56:06.445705 1483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:06.465342 1483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:06.469393 1483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:06.469688 1483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:06.488809 1483 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:34641
I20260925 14:56:06.488830 1550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:34641 every 8 connection(s)
I20260925 14:56:06.490406 1483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:06.497519 1551 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:56:06.498342 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 1483
I20260925 14:56:06.499168 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:38953
--local_ip_for_outbound_sockets=127.17.133.193
--tserver_master_addrs=127.17.133.254:34641
--webserver_port=38097
--webserver_interface=127.17.133.193
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:06.501030 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Bootstrap starting.
I20260925 14:56:06.527621 1551 log.cc:824] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:06.542902 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:56:06.543526 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Bootstrap complete.
I20260925 14:56:06.557257 1551 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:56:06.558497 1551 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6040f5d1d2a46c399e645c2c6e1226c, State: Initialized, Role: FOLLOWER
I20260925 14:56:06.559454 1551 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:56:06.559830 1551 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:06.559978 1551 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:06.560158 1551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:56:06.566214 1551 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:56:06.567014 1551 leader_election.cc:302] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6040f5d1d2a46c399e645c2c6e1226c; no voters:
I20260925 14:56:06.567737 1551 leader_election.cc:288] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:56:06.567848 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:56:06.569198 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 2 LEADER]: Becoming Leader. State: Replica: d6040f5d1d2a46c399e645c2c6e1226c, State: Running, Role: LEADER
I20260925 14:56:06.569808 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:56:06.570206 1551 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:06.572397 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } } }
I20260925 14:56:06.572391 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6040f5d1d2a46c399e645c2c6e1226c. Latest consensus state: current_term: 2 leader_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } } }
I20260925 14:56:06.572840 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:06.572844 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:06.575377 1561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:06.583675 1561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:06.585134 1561 catalog_manager.cc:1259] Loaded cluster ID: e2e2adb6bca74e5695fcbbba895f218d
I20260925 14:56:06.585284 1561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:06.589334 1561 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:06.590979 1561 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Loaded TSK: 0
I20260925 14:56:06.629958 1561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Generated new TSK 1
I20260925 14:56:06.630424 1561 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:56:06.857962 1553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:06.858294 1553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:06.858451 1553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:06.865492 1553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:06.865629 1553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:06.865792 1553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:56:06.875332 1553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:38953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=38097
--tserver_master_addrs=127.17.133.254:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:06.876766 1553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:06.878170 1553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:06.892238 1579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:06.892298 1578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:06.893361 1581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:06.894339 1553 server_base.cc:1034] running on GCE node
I20260925 14:56:06.894956 1553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:06.895849 1553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:06.897060 1553 hybrid_clock.cc:648] HybridClock initialized: now 1790348166897009 us; error 48 us; skew 500 ppm
I20260925 14:56:06.897367 1553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:06 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348166, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:06.901377 1553 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:56:06.914366 1553 webserver.cc:460] Webserver started at http://127.17.133.193:38097/ using document root <none> and password file <none>
I20260925 14:56:06.915551 1553 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:06.915724 1553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:06.924453 1553 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 14:56:06.928246 1588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:06.930140 1553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 14:56:06.930404 1553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "d7588880f61a4cd5bf8b626a87356f92"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "0fb961a42437692b03472284c0fcc18f"
server_key_iv: "d04100795e185bf9c3fe259a63ed1683"
server_key_version: "encryptionkey@0"
I20260925 14:56:06.931787 1553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:06.956543 1553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:06.961290 1553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:06.961566 1553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:06.962522 1553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:06.963819 1553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:06.963938 1553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:06.964071 1553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:06.964155 1553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:07.003887 1553 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:38953
I20260925 14:56:07.003939 1700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:38953 every 8 connection(s)
I20260925 14:56:07.005426 1553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:07.006806 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 1553
I20260925 14:56:07.007606 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:43323
--local_ip_for_outbound_sockets=127.17.133.194
--tserver_master_addrs=127.17.133.254:34641
--webserver_port=33943
--webserver_interface=127.17.133.194
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:56:07 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348166, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:07.033283 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.008746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:58319)
0925 14:56:07.009607 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:07.009655 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:07.012816 (+ 3161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:07.014477 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:07.014484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:07.015183 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:07.015931 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.015955 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.018362 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.018366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.019585 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.019596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.021058 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.021062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.021263 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.021931 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:07.021953 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:07.024110 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:07.029107 (+ 4997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:07.029140 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:07.029152 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:07.029592 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:07.030126 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:07.030129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:07.030131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:07.030305 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:07.031282 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:07.031288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:07.031587 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 14:56:07.031816 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 14:56:07.032212 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":497,"threads_started":1}
I20260925 14:56:07.033946 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.009735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:58319 (local address 127.17.133.254:34641)
0925 14:56:07.013146 (+ 3411us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.013164 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.013213 (+ 49us) server_negotiation.cc:407] Connection header received
0925 14:56:07.013557 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.013576 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.014008 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.014372 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:07.016162 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.018210 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.019726 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.020873 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.021050 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.024404 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:07.024417 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:07.024439 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:07.024491 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:07.028933 (+ 4442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.029739 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.029749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.029760 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.029996 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.030416 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.030424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.030428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.031135 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:07.031347 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:07.032118 (+ 771us) server_negotiation.cc:299] Negotiation successful
0925 14:56:07.032609 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3168,"thread_start_us":3037,"threads_started":1}
I20260925 14:56:07.036475 1701 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34641
I20260925 14:56:07.036893 1701 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:07.037752 1701 heartbeater.cc:511] Master 127.17.133.254:34641 requested a full tablet report, sending...
I20260925 14:56:07.039899 1516 ts_manager.cc:194] Registered new tserver with Master: d7588880f61a4cd5bf8b626a87356f92 (127.17.133.193:38953)
I20260925 14:56:07.041388 1516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:58319
W20260925 14:56:07.326925 1705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:07.327162 1705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:07.327293 1705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:07.334326 1705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:07.334424 1705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:07.334607 1705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:56:07.343794 1705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:43323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=33943
--tserver_master_addrs=127.17.133.254:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:07.345062 1705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:07.346259 1705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:07.358924 1710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:07.358989 1713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:07.358959 1711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:07.359788 1705 server_base.cc:1034] running on GCE node
I20260925 14:56:07.360303 1705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:07.361268 1705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:07.362458 1705 hybrid_clock.cc:648] HybridClock initialized: now 1790348167362419 us; error 32 us; skew 500 ppm
I20260925 14:56:07.362732 1705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:07 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348167, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:07.365687 1705 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:56:07.366991 1705 webserver.cc:460] Webserver started at http://127.17.133.194:33943/ using document root <none> and password file <none>
I20260925 14:56:07.367589 1705 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:07.367705 1705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:07.372818 1705 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:07.375314 1720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:07.376698 1705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:07.376828 1705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "c30625681fe747e0b00297efd5716c57"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "b0b2e3ebd03abe46e978084dfe7e4bca"
server_key_iv: "f517477017e6a15510bd0c4e847ce641"
server_key_version: "encryptionkey@0"
I20260925 14:56:07.377758 1705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:07.391958 1705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:07.395707 1705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:07.395926 1705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:07.396835 1705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:07.398061 1705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:07.398147 1705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:07.398226 1705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:07.398276 1705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:07.430087 1705 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:43323
I20260925 14:56:07.430171 1832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:43323 every 8 connection(s)
I20260925 14:56:07.431463 1705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:07.432495 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 1705
I20260925 14:56:07.433244 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:45953
--local_ip_for_outbound_sockets=127.17.133.195
--tserver_master_addrs=127.17.133.254:34641
--webserver_port=42373
--webserver_interface=127.17.133.195
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:56:07 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348167, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:07.454656 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.434282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:53735 (local address 127.17.133.254:34641)
0925 14:56:07.434507 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.434515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.435746 (+ 1231us) server_negotiation.cc:407] Connection header received
0925 14:56:07.438554 (+ 2808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.438558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.438670 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.438767 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:07.440804 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.441471 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.443431 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.444240 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.444393 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.447978 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:07.447988 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:07.447995 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:07.448040 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:07.451411 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.452075 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.452084 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.452090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.452243 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.452731 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.452739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.452745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.453149 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:07.453321 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:07.453923 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 14:56:07.454191 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:56:07.455164 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.434764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:53735)
0925 14:56:07.435579 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:07.435624 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:07.438262 (+ 2638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:07.439032 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:07.439046 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:07.439828 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:07.440641 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.440668 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.441622 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.441629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.443286 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.443297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.444453 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.444458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.444724 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.445432 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:07.445453 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:07.447765 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:07.451529 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:07.451534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:07.451545 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:07.451925 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:07.452348 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:07.452351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:07.452353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:07.452571 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:07.453273 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:07.453279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:07.453563 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 14:56:07.453768 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 14:56:07.454108 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":298,"threads_started":1}
I20260925 14:56:07.456622 1833 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34641
I20260925 14:56:07.456914 1833 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:07.457569 1833 heartbeater.cc:511] Master 127.17.133.254:34641 requested a full tablet report, sending...
I20260925 14:56:07.459008 1516 ts_manager.cc:194] Registered new tserver with Master: c30625681fe747e0b00297efd5716c57 (127.17.133.194:43323)
I20260925 14:56:07.459774 1516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:53735
W20260925 14:56:07.755228 1836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:07.755496 1836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:07.755686 1836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:07.763665 1836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:07.763808 1836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:07.763984 1836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:56:07.774006 1836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:45953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=42373
--tserver_master_addrs=127.17.133.254:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:07.775369 1836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:07.776800 1836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:07.789626 1841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:07.789661 1842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:07.789970 1844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:07.790288 1836 server_base.cc:1034] running on GCE node
I20260925 14:56:07.790815 1836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:07.791649 1836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:07.792882 1836 hybrid_clock.cc:648] HybridClock initialized: now 1790348167792821 us; error 55 us; skew 500 ppm
I20260925 14:56:07.793181 1836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:07 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348167, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:07.796001 1836 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:56:07.797324 1836 webserver.cc:460] Webserver started at http://127.17.133.195:42373/ using document root <none> and password file <none>
I20260925 14:56:07.798000 1836 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:07.798141 1836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:07.803084 1836 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 14:56:07.805522 1851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:07.806654 1836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:56:07.806819 1836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "a37d21a350aa4bd8837e75f19d6ea917"
format_stamp: "Formatted at 2026-09-25 14:55:45 on dist-test-slave-v5vl"
server_key: "a0606ff8447def512c36b0cb6aa435bd"
server_key_iv: "02f9cedccde45e13fcd1c50bad2c4fa8"
server_key_version: "encryptionkey@0"
I20260925 14:56:07.807709 1836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:07.828311 1836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:07.832084 1836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:07.832329 1836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:07.833297 1836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:07.834522 1836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:07.834637 1836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:07.834779 1836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:07.834872 1836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:07.866307 1836 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:45953
I20260925 14:56:07.866385 1963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:45953 every 8 connection(s)
I20260925 14:56:07.867666 1836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:07.869207 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 1836
Sep 25 14:56:07 dist-test-slave-v5vl krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348167, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:07.884865 1967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.870276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.17.133.254:34641)
0925 14:56:07.871465 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.871469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.871483 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:56:07.871548 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.871551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.871604 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.871805 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:07.873543 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.874462 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.875906 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.876385 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.876571 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.879308 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:07.879313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:07.879316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:07.879344 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:07.882129 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.882782 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.882789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.882797 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.882930 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.883299 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.883304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.883309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.883564 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:07.883877 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:07.884189 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 14:56:07.884429 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1111,"thread_start_us":1006,"threads_started":1}
I20260925 14:56:07.888237 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.870204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:57933 (local address 127.17.133.254:34641)
0925 14:56:07.870424 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.870433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.872141 (+ 1708us) server_negotiation.cc:407] Connection header received
0925 14:56:07.873993 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.873998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.874061 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.874160 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:07.875671 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.876570 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.878350 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.878781 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.878909 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.882536 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:07.882542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:07.882546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:07.882583 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:07.885660 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.886447 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.886450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.886452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.886550 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.886965 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.886968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.886970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.887247 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:07.887340 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:07.887897 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 14:56:07.888029 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:56:07.889149 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.870963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:57933)
0925 14:56:07.871967 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:07.872010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:07.873760 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:07.874328 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:07.874336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:07.874981 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:07.875511 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.875528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.876763 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.876767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.878212 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.878223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.879004 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.879011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.879220 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.879881 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:07.879919 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:07.882352 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:07.885799 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:07.885807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:07.885821 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:07.886352 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:07.886664 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:07.886668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:07.886671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:07.886875 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:07.887349 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:07.887356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:07.887637 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 14:56:07.887841 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 14:56:07.888144 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":425,"threads_started":1}
I20260925 14:56:07.890290 1964 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34641
I20260925 14:56:07.890565 1964 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:07.891192 1964 heartbeater.cc:511] Master 127.17.133.254:34641 requested a full tablet report, sending...
I20260925 14:56:07.892432 1516 ts_manager.cc:194] Registered new tserver with Master: a37d21a350aa4bd8837e75f19d6ea917 (127.17.133.195:45953)
I20260925 14:56:07.893317 1516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:57933
I20260925 14:56:07.899091 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:07.904711 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.899955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.17.133.254:34641)
0925 14:56:07.900147 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.900153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.900371 (+ 218us) server_negotiation.cc:407] Connection header received
0925 14:56:07.900516 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.900519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.900697 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.900802 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:07.901151 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.901830 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.903232 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.903588 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.903672 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.904096 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:07.904267 (+ 171us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 14:56:07.904493 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47490: token expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 14:56:07.904838 1704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47490: token expired
W20260925 14:56:07.905354 1438 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.133.254:34641)
I20260925 14:56:07.916442 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.905701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.17.133.254:34641)
0925 14:56:07.905838 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.905842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.905855 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:56:07.905932 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.905934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.905977 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.906043 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:07.907242 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.907899 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.909212 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.909558 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.909639 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.912129 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:07.912134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:07.912137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:07.912163 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:07.914445 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.915007 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.915010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.915012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.915121 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.915414 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.915417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.915419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.915674 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:07.915856 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:07.916079 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 14:56:07.916224 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:56:07.919504 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:07.926932 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:07.960863 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.939101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.17.133.193:38953)
0925 14:56:07.951694 (+ 12593us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.951706 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.952609 (+ 903us) server_negotiation.cc:407] Connection header received
0925 14:56:07.952994 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.953000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.953195 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.953395 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:07.954092 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.955653 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.958249 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.959658 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.959851 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.960033 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:07.960348 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 14:56:07.960506 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12394,"thread_start_us":12260,"threads_started":1}
I20260925 14:56:07.960901 1974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.938789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:38953 (local address 127.0.0.1:48474)
0925 14:56:07.952388 (+ 13599us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:07.952480 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:07.952699 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:07.953391 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:07.953398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:07.953425 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:07.953835 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.953845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.955819 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.955825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.958064 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.958077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.959897 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.959904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.960114 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.960154 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 14:56:07.960298 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 14:56:07.960478 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13387,"spinlock_wait_cycles":55552,"thread_start_us":13128,"threads_started":1}
I20260925 14:56:07.962458 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.938554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:45953 (local address 127.0.0.1:47406)
0925 14:56:07.952388 (+ 13834us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:07.952464 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:07.952700 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:07.953812 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:07.953823 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:07.953890 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:07.954317 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.954330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.956241 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.956248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.959427 (+ 3179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.959442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.961539 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.961547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.961722 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.961746 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 14:56:07.961852 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:56:07.962002 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13556,"spinlock_wait_cycles":11008,"thread_start_us":13833,"threads_started":1}
I20260925 14:56:07.963289 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.938809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.17.133.195:45953)
0925 14:56:07.951161 (+ 12352us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.951171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.952696 (+ 1525us) server_negotiation.cc:407] Connection header received
0925 14:56:07.952990 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.953002 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.953258 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.953461 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:07.954505 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.956081 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.959647 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.961079 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.961265 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.961414 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:07.962889 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0925 14:56:07.963045 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12151,"thread_start_us":13897,"threads_started":1}
I20260925 14:56:07.966048 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.954625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:43323 (local address 127.0.0.1:50068)
0925 14:56:07.958570 (+ 3945us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:07.958609 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:07.958740 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:07.959369 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:07.959374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:07.959400 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:07.959789 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.959801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.961837 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.961841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.963896 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:07.963911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.965307 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.965312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:07.965487 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.965512 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 14:56:07.965618 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:56:07.965739 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3772,"thread_start_us":3614,"threads_started":1}
I20260925 14:56:07.965528 1636 tablet_service.cc:1467] Processing CreateTablet for tablet 146244e878ad41ceb375b2e74a1adffe (DEFAULT_TABLE table=test-table [id=9bcc838261f44bd9aa5e4b823537db8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:07.966104 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.953153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.17.133.194:43323)
0925 14:56:07.957840 (+ 4687us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.957855 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.958733 (+ 878us) server_negotiation.cc:407] Connection header received
0925 14:56:07.958883 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.958891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.959149 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.959352 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:07.959976 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.961510 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.964065 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.965095 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.965276 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.965436 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:07.965666 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 14:56:07.965821 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4548,"thread_start_us":4457,"threads_started":1}
I20260925 14:56:07.966974 1636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146244e878ad41ceb375b2e74a1adffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:07.966809 1899 tablet_service.cc:1467] Processing CreateTablet for tablet 146244e878ad41ceb375b2e74a1adffe (DEFAULT_TABLE table=test-table [id=9bcc838261f44bd9aa5e4b823537db8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:07.968149 1899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146244e878ad41ceb375b2e74a1adffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:07.971619 1768 tablet_service.cc:1467] Processing CreateTablet for tablet 146244e878ad41ceb375b2e74a1adffe (DEFAULT_TABLE table=test-table [id=9bcc838261f44bd9aa5e4b823537db8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:07.972633 1768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146244e878ad41ceb375b2e74a1adffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:07.982609 1980 tablet_bootstrap.cc:492] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: Bootstrap starting.
I20260925 14:56:07.987193 1980 tablet_bootstrap.cc:654] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:07.988160 1982 tablet_bootstrap.cc:492] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: Bootstrap starting.
I20260925 14:56:07.989302 1981 tablet_bootstrap.cc:492] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: Bootstrap starting.
I20260925 14:56:07.991176 1982 tablet_bootstrap.cc:654] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:07.991881 1981 tablet_bootstrap.cc:654] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:07.992161 1982 log.cc:824] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:07.992951 1981 log.cc:824] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:07.993266 1980 log.cc:824] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:07.994979 1982 tablet_bootstrap.cc:492] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: No bootstrap required, opened a new log
I20260925 14:56:07.995240 1982 ts_tablet_manager.cc:1397] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 14:56:07.996097 1980 tablet_bootstrap.cc:492] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: No bootstrap required, opened a new log
I20260925 14:56:07.996351 1980 ts_tablet_manager.cc:1397] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260925 14:56:07.996486 1981 tablet_bootstrap.cc:492] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: No bootstrap required, opened a new log
I20260925 14:56:07.996771 1981 ts_tablet_manager.cc:1397] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 14:56:08.004027 1980 raft_consensus.cc:348] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } }
I20260925 14:56:08.004613 1980 raft_consensus.cc:374] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:08.004729 1980 raft_consensus.cc:729] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7588880f61a4cd5bf8b626a87356f92, State: Initialized, Role: FOLLOWER
I20260925 14:56:08.005543 1980 consensus_queue.cc:260] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } }
I20260925 14:56:08.005823 1982 raft_consensus.cc:348] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } }
I20260925 14:56:08.006304 1982 raft_consensus.cc:374] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:08.006419 1982 raft_consensus.cc:729] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30625681fe747e0b00297efd5716c57, State: Initialized, Role: FOLLOWER
I20260925 14:56:08.007019 1982 consensus_queue.cc:260] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } }
I20260925 14:56:08.007185 1981 raft_consensus.cc:348] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } }
I20260925 14:56:08.007670 1981 raft_consensus.cc:374] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:08.007858 1981 raft_consensus.cc:729] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37d21a350aa4bd8837e75f19d6ea917, State: Initialized, Role: FOLLOWER
I20260925 14:56:08.008769 1981 consensus_queue.cc:260] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } }
I20260925 14:56:08.014761 1833 heartbeater.cc:503] Master 127.17.133.254:34641 was elected leader, sending a full tablet report...
I20260925 14:56:08.015766 1982 ts_tablet_manager.cc:1428] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: Time spent starting tablet: real 0.020s user 0.014s sys 0.006s
I20260925 14:56:08.025287 1701 heartbeater.cc:503] Master 127.17.133.254:34641 was elected leader, sending a full tablet report...
I20260925 14:56:08.026005 1980 ts_tablet_manager.cc:1428] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: Time spent starting tablet: real 0.030s user 0.005s sys 0.025s
I20260925 14:56:08.026037 1981 ts_tablet_manager.cc:1428] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: Time spent starting tablet: real 0.029s user 0.019s sys 0.007s
I20260925 14:56:08.026880 1964 heartbeater.cc:503] Master 127.17.133.254:34641 was elected leader, sending a full tablet report...
W20260925 14:56:08.119300 1965 tablet.cc:2369] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:56:08.183513 1834 tablet.cc:2369] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:56:08.257797 1702 tablet.cc:2369] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:08.280547 1988 raft_consensus.cc:482] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:08.280907 1988 raft_consensus.cc:504] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } }
I20260925 14:56:08.282716 1988 leader_election.cc:288] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7588880f61a4cd5bf8b626a87356f92 (127.17.133.193:38953), c30625681fe747e0b00297efd5716c57 (127.17.133.194:43323)
I20260925 14:56:08.288388 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.283156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:38953 (local address 127.17.133.195:45373)
0925 14:56:08.283383 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.283403 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.283540 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.283938 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.283943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.283985 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:08.284168 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.284176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.285323 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.285332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.286631 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.286643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.287872 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.287876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.288009 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.288032 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.288105 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.288200 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 14:56:08.288535 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.283135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:45373 (local address 127.17.133.193:38953)
0925 14:56:08.283402 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.283408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.283575 (+ 167us) server_negotiation.cc:407] Connection header received
0925 14:56:08.283662 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.283668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.283781 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.283881 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:08.284376 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.285195 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.286796 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.287739 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.287891 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.287966 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.288156 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.288286 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 14:56:08.289549 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.283319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:40311 (local address 127.17.133.194:43323)
0925 14:56:08.283520 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.283526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.284238 (+ 712us) server_negotiation.cc:407] Connection header received
0925 14:56:08.284485 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.284489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.284546 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.284671 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:08.285172 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.285849 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.287488 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.288719 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.288882 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.288952 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.289169 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.289299 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:56:08.289613 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.283223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:43323 (local address 127.17.133.195:40311)
0925 14:56:08.284143 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.284156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.284343 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.284687 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.284691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.284714 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:08.285014 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.285021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.285983 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.285988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.287307 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.287318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.288902 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.288906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.289016 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.289038 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.289139 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.289251 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":198,"thread_start_us":373,"threads_started":1}
I20260925 14:56:08.289474 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146244e878ad41ceb375b2e74a1adffe" candidate_uuid: "a37d21a350aa4bd8837e75f19d6ea917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7588880f61a4cd5bf8b626a87356f92" is_pre_election: true
I20260925 14:56:08.290052 1656 raft_consensus.cc:2454] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a37d21a350aa4bd8837e75f19d6ea917 in term 0.
I20260925 14:56:08.290632 1788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146244e878ad41ceb375b2e74a1adffe" candidate_uuid: "a37d21a350aa4bd8837e75f19d6ea917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30625681fe747e0b00297efd5716c57" is_pre_election: true
I20260925 14:56:08.290831 1853 leader_election.cc:302] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a37d21a350aa4bd8837e75f19d6ea917, d7588880f61a4cd5bf8b626a87356f92; no voters:
I20260925 14:56:08.291147 1788 raft_consensus.cc:2454] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a37d21a350aa4bd8837e75f19d6ea917 in term 0.
I20260925 14:56:08.291247 1988 raft_consensus.cc:2781] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:08.291409 1988 raft_consensus.cc:482] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:08.291512 1988 raft_consensus.cc:3037] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:08.293735 1988 raft_consensus.cc:504] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } }
I20260925 14:56:08.294301 1988 leader_election.cc:288] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [CANDIDATE]: Term 1 election: Requested vote from peers d7588880f61a4cd5bf8b626a87356f92 (127.17.133.193:38953), c30625681fe747e0b00297efd5716c57 (127.17.133.194:43323)
I20260925 14:56:08.294669 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146244e878ad41ceb375b2e74a1adffe" candidate_uuid: "a37d21a350aa4bd8837e75f19d6ea917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7588880f61a4cd5bf8b626a87356f92"
I20260925 14:56:08.294714 1788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146244e878ad41ceb375b2e74a1adffe" candidate_uuid: "a37d21a350aa4bd8837e75f19d6ea917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30625681fe747e0b00297efd5716c57"
I20260925 14:56:08.294873 1656 raft_consensus.cc:3037] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:08.294917 1788 raft_consensus.cc:3037] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:08.297041 1656 raft_consensus.cc:2454] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a37d21a350aa4bd8837e75f19d6ea917 in term 1.
I20260925 14:56:08.297108 1788 raft_consensus.cc:2454] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a37d21a350aa4bd8837e75f19d6ea917 in term 1.
I20260925 14:56:08.297474 1853 leader_election.cc:302] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a37d21a350aa4bd8837e75f19d6ea917, d7588880f61a4cd5bf8b626a87356f92; no voters:
I20260925 14:56:08.297772 1988 raft_consensus.cc:2781] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:08.298183 1988 raft_consensus.cc:686] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 1 LEADER]: Becoming Leader. State: Replica: a37d21a350aa4bd8837e75f19d6ea917, State: Running, Role: LEADER
I20260925 14:56:08.298640 1988 consensus_queue.cc:237] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } }
I20260925 14:56:08.303205 1516 catalog_manager.cc:5495] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 reported cstate change: term changed from 0 to 1, leader changed from <none> to a37d21a350aa4bd8837e75f19d6ea917 (127.17.133.195). New cstate: current_term: 1 leader_uuid: "a37d21a350aa4bd8837e75f19d6ea917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a37d21a350aa4bd8837e75f19d6ea917" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 45953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 } health_report { overall_health: UNKNOWN } } }
I20260925 14:56:08.406960 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.399785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.17.133.195:45953)
0925 14:56:08.399969 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.399973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.399989 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:56:08.400076 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.400080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.400169 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.400332 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:08.400855 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.401802 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.403983 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.404655 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.404805 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.405537 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:08.406296 (+ 759us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:08.406459 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.406525 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.406630 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:56:08.414747 1788 raft_consensus.cc:1261] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [term 1 FOLLOWER]: Refusing update from remote peer a37d21a350aa4bd8837e75f19d6ea917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:08.414925 1656 raft_consensus.cc:1261] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [term 1 FOLLOWER]: Refusing update from remote peer a37d21a350aa4bd8837e75f19d6ea917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:08.415648 1990 consensus_queue.cc:1035] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30625681fe747e0b00297efd5716c57" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 43323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:08.416057 1988 consensus_queue.cc:1035] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7588880f61a4cd5bf8b626a87356f92" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:08.426736 1993 mvcc.cc:205] Tried to move back new op lower bound from 7333266097817767936 to 7333266097354444800. Current Snapshot: MvccSnapshot[applied={T|T < 7333266097817767936}]
I20260925 14:56:08.431772 1995 mvcc.cc:205] Tried to move back new op lower bound from 7333266097817767936 to 7333266097354444800. Current Snapshot: MvccSnapshot[applied={T|T < 7333266097817767936}]
I20260925 14:56:08.447983 1515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:08.448233 1515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:08.451864 1515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Sending DeleteTablet for 3 replicas of tablet 146244e878ad41ceb375b2e74a1adffe
I20260925 14:56:08.452927 1636 tablet_service.cc:1514] Processing DeleteTablet for tablet 146244e878ad41ceb375b2e74a1adffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:08 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:48474
I20260925 14:56:08.452950 1899 tablet_service.cc:1514] Processing DeleteTablet for tablet 146244e878ad41ceb375b2e74a1adffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:08 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:47406
I20260925 14:56:08.453783 2003 tablet_replica.cc:330] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: stopping tablet replica
I20260925 14:56:08.454133 1768 tablet_service.cc:1514] Processing DeleteTablet for tablet 146244e878ad41ceb375b2e74a1adffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:08 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:50068
I20260925 14:56:08.454249 2003 raft_consensus.cc:2229] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:08.454731 2003 raft_consensus.cc:2258] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:08.455775 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 1553
I20260925 14:56:08.457296 2005 tablet_replica.cc:330] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: stopping tablet replica
I20260925 14:56:08.457832 2005 raft_consensus.cc:2229] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:08.458489 2005 raft_consensus.cc:2258] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:08.460726 2003 ts_tablet_manager.cc:1905] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:08.460754 2005 ts_tablet_manager.cc:1905] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:08.469660 2005 ts_tablet_manager.cc:1918] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:08.469870 2005 log.cc:1196] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/146244e878ad41ceb375b2e74a1adffe
I20260925 14:56:08.470520 2005 ts_tablet_manager.cc:1939] T 146244e878ad41ceb375b2e74a1adffe P c30625681fe747e0b00297efd5716c57: Deleting consensus metadata
I20260925 14:56:08.472543 1503 catalog_manager.cc:4841] TS c30625681fe747e0b00297efd5716c57 (127.17.133.194:43323): tablet 146244e878ad41ceb375b2e74a1adffe (table test-table [id=9bcc838261f44bd9aa5e4b823537db8b]) successfully deleted
I20260925 14:56:08.472609 2003 ts_tablet_manager.cc:1918] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:08.472764 2003 log.cc:1196] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/146244e878ad41ceb375b2e74a1adffe
I20260925 14:56:08.473405 2003 ts_tablet_manager.cc:1939] T 146244e878ad41ceb375b2e74a1adffe P d7588880f61a4cd5bf8b626a87356f92: Deleting consensus metadata
I20260925 14:56:08.475050 1501 catalog_manager.cc:4841] TS d7588880f61a4cd5bf8b626a87356f92 (127.17.133.193:38953): tablet 146244e878ad41ceb375b2e74a1adffe (table test-table [id=9bcc838261f44bd9aa5e4b823537db8b]) successfully deleted
I20260925 14:56:08.477175 2004 tablet_replica.cc:330] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: stopping tablet replica
I20260925 14:56:08.477562 2004 raft_consensus.cc:2229] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:08.478173 2004 raft_consensus.cc:2258] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:08.480216 2004 ts_tablet_manager.cc:1905] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:08.484427 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.456746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.17.133.193:38953)
0925 14:56:08.456907 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.456913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.469405 (+ 12492us) server_negotiation.cc:407] Connection header received
0925 14:56:08.469851 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.469857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.469941 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.470023 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:08.472090 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.473193 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.474550 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.475117 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.475262 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.478256 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:08.478270 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:08.478284 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:08.478323 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:08.481463 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.482299 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.482307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.482310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.482461 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.482822 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.482827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.482831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.483193 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:08.483538 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.483748 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.483998 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:56:08.485134 1696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:08.487291 2004 ts_tablet_manager.cc:1918] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:08.487501 2004 log.cc:1196] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/146244e878ad41ceb375b2e74a1adffe
I20260925 14:56:08.488148 2004 ts_tablet_manager.cc:1939] T 146244e878ad41ceb375b2e74a1adffe P a37d21a350aa4bd8837e75f19d6ea917: Deleting consensus metadata
I20260925 14:56:08.489692 1503 catalog_manager.cc:4841] TS a37d21a350aa4bd8837e75f19d6ea917 (127.17.133.195:45953): tablet 146244e878ad41ceb375b2e74a1adffe (table test-table [id=9bcc838261f44bd9aa5e4b823537db8b]) successfully deleted
I20260925 14:56:08.596688 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 1553
I20260925 14:56:08.614298 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 1705
I20260925 14:56:08.629756 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.615099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.17.133.194:43323)
0925 14:56:08.615330 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.615337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.615361 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:56:08.615436 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.615443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.615523 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.615635 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:08.616953 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.618071 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.619412 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.620068 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.620198 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.622823 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:08.622841 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:08.622869 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:08.622915 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:08.626846 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.627507 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.627519 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.627522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.627699 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.628075 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.628080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.628084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.628534 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:08.628992 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.629147 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.629376 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 14:56:08.630525 1828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:08.719096 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 1705
I20260925 14:56:08.736871 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 1836
I20260925 14:56:08.751514 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.737675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.17.133.195:45953)
0925 14:56:08.737850 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.737858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.737881 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:56:08.738103 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.738107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.738188 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.738311 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:08.739594 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.740422 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.741949 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.742398 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.742541 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.745188 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:08.745200 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:08.745215 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:08.745263 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:08.748450 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.749271 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.749287 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.749290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.749507 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.749923 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.749928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.749930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.750280 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:08.750586 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.750904 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.751137 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:56:08.752467 1959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:08.873553 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 1836
I20260925 14:56:08.893337 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 1483
I20260925 14:56:08.894492 1546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:09.008774 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 1483
I20260925 14:56:29.030330 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34641
--webserver_interface=127.17.133.254
--webserver_port=38635
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:34641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:29.352089 2011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:29.352360 2011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:29.352478 2011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:29.359567 2011 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:56:29.359652 2011 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:56:29.359726 2011 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:56:29.359812 2011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:29.359910 2011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:29.360028 2011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:29.360111 2011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:29.360203 2011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:29.369643 2011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:34641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:29.370890 2011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:29.372155 2011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:29.383874 2016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:29.383836 2017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:29.383912 2019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:29.384505 2011 server_base.cc:1034] running on GCE node
I20260925 14:56:29.385411 2011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:29.386806 2011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:29.388029 2011 hybrid_clock.cc:648] HybridClock initialized: now 1790348189387971 us; error 34 us; skew 500 ppm
I20260925 14:56:29.388315 2011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:29 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348189, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:29.395133 2011 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:56:29.396469 2011 webserver.cc:460] Webserver started at http://127.17.133.254:38635/ using document root <none> and password file <none>
I20260925 14:56:29.397219 2011 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:29.397306 2011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:29.402493 2011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 14:56:29.405347 2026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:29.406694 2011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:29.406832 2011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "d6040f5d1d2a46c399e645c2c6e1226c"
format_stamp: "Formatted at 2026-09-25 14:55:43 on dist-test-slave-v5vl"
server_key: "105bbf78dca52c210c8b1d427da39f32"
server_key_iv: "0182284bdaeb18addb2b44e543c47437"
server_key_version: "encryptionkey@0"
I20260925 14:56:29.407740 2011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:29.427835 2011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:29.431444 2011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:29.431664 2011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:29.449909 2011 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:34641
I20260925 14:56:29.449939 2077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:34641 every 8 connection(s)
I20260925 14:56:29.451447 2011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:29.452893 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 2011
I20260925 14:56:29.453923 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:38953
--local_ip_for_outbound_sockets=127.17.133.193
--tserver_master_addrs=127.17.133.254:34641
--webserver_port=38097
--webserver_interface=127.17.133.193
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:29.461323 2078 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:56:29.465140 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Bootstrap starting.
I20260925 14:56:29.485284 2078 log.cc:824] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:29.500651 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:56:29.501227 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Bootstrap complete.
I20260925 14:56:29.510295 2078 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:56:29.511353 2078 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6040f5d1d2a46c399e645c2c6e1226c, State: Initialized, Role: FOLLOWER
I20260925 14:56:29.511986 2078 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:56:29.512233 2078 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:29.512367 2078 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:29.512568 2078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:56:29.516335 2078 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:56:29.516839 2078 leader_election.cc:302] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6040f5d1d2a46c399e645c2c6e1226c; no voters:
I20260925 14:56:29.517534 2078 leader_election.cc:288] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:56:29.517683 2082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:56:29.518854 2082 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [term 3 LEADER]: Becoming Leader. State: Replica: d6040f5d1d2a46c399e645c2c6e1226c, State: Running, Role: LEADER
I20260925 14:56:29.519438 2082 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } }
I20260925 14:56:29.519760 2078 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:29.521808 2084 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6040f5d1d2a46c399e645c2c6e1226c. Latest consensus state: current_term: 3 leader_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } } }
I20260925 14:56:29.522230 2084 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:29.522832 2083 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6040f5d1d2a46c399e645c2c6e1226c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34641 } } }
I20260925 14:56:29.523175 2083 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:29.525347 2088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:29.535454 2088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:29.537364 2088 catalog_manager.cc:1259] Loaded cluster ID: e2e2adb6bca74e5695fcbbba895f218d
I20260925 14:56:29.537540 2088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:29.542908 2088 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:29.544998 2088 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Loaded TSK: 0
I20260925 14:56:29.545252 2088 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6040f5d1d2a46c399e645c2c6e1226c: Loaded TSK: 1
F20260925 14:56:29.546693 2088 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb4c79fd9ad google::LogMessage::Fail() at ??:0
@ 0x7fb4c7a0088a google::LogMessage::SendToLog() at ??:0
@ 0x7fb4c79fd4de google::LogMessage::Flush() at ??:0
@ 0x7fb4c79fea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb4cb7617d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb4cb75073e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb4dd42c1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb4dd42b783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb4dd4b72a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb4dd4b70c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb4dd5015e9 std::function<>::operator()() at ??:0
@ 0x7fb4dd42fcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb4dd42dc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb4dd4b5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb4dd4b5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb4ca649345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb4ca64c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb4ca64c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb4ca627eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb4cd9016db start_thread at ??:0
@ 0x7fb4c689761f clone at ??:0
W20260925 14:56:29.831880 2080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:29.832175 2080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:29.832335 2080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:29.839690 2080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:29.839795 2080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:29.840000 2080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:56:29.849726 2080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:38953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=38097
--tserver_master_addrs=127.17.133.254:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:29.850929 2080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:29.852169 2080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:29.865085 2129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:29.865154 2126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:29.865128 2127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:29.866151 2080 server_base.cc:1034] running on GCE node
I20260925 14:56:29.866653 2080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:29.867508 2080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:29.868726 2080 hybrid_clock.cc:648] HybridClock initialized: now 1790348189868647 us; error 54 us; skew 500 ppm
I20260925 14:56:29.869051 2080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:29 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348189, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:29.872035 2080 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:56:29.873435 2080 webserver.cc:460] Webserver started at http://127.17.133.193:38097/ using document root <none> and password file <none>
I20260925 14:56:29.874078 2080 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:29.874172 2080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:29.879209 2080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 14:56:29.881752 2136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:29.883004 2080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:29.883133 2080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "d7588880f61a4cd5bf8b626a87356f92"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "0fb961a42437692b03472284c0fcc18f"
server_key_iv: "d04100795e185bf9c3fe259a63ed1683"
server_key_version: "encryptionkey@0"
I20260925 14:56:29.884106 2080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:29.901899 2080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:29.905504 2080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:29.905722 2080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:29.906554 2080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:29.907744 2080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:29.907826 2080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:29.907904 2080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:29.907958 2080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:29.941033 2080 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:38953
I20260925 14:56:29.941073 2248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:38953 every 8 connection(s)
I20260925 14:56:29.942390 2080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:29.946929 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:29.945164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:37153)
0925 14:56:29.945859 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:29.946118 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":265,"threads_started":1}
W20260925 14:56:29.947573 2249 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:34641 (0 consecutive failures): Network error: Failed to ping master at 127.17.133.254:34641: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
I20260925 14:56:29.948292 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:29.948022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:41629)
0925 14:56:29.948140 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:29.948200 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:56:29.949036 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:29.948767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:57829)
0925 14:56:29.948872 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:29.948938 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 14:56:29.949270 2249 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:56:29.949939 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 2080
I20260925 14:56:29.950523 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:43323
--local_ip_for_outbound_sockets=127.17.133.194
--tserver_master_addrs=127.17.133.254:34641
--webserver_port=33943
--webserver_interface=127.17.133.194
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:30.274458 2252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:30.274767 2252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:30.274863 2252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:30.281880 2252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:30.281973 2252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:30.282095 2252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:56:30.292506 2252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:43323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=33943
--tserver_master_addrs=127.17.133.254:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:30.293672 2252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:30.294870 2252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:30.307390 2257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:30.307447 2258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:30.307479 2260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:30.308288 2252 server_base.cc:1034] running on GCE node
I20260925 14:56:30.308876 2252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:30.309731 2252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:30.310961 2252 hybrid_clock.cc:648] HybridClock initialized: now 1790348190310922 us; error 51 us; skew 500 ppm
I20260925 14:56:30.311282 2252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:30 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348190, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:30.314347 2252 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:56:30.315724 2252 webserver.cc:460] Webserver started at http://127.17.133.194:33943/ using document root <none> and password file <none>
I20260925 14:56:30.316399 2252 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:30.316551 2252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:30.321696 2252 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 14:56:30.324218 2267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:30.325542 2252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:56:30.325724 2252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "c30625681fe747e0b00297efd5716c57"
format_stamp: "Formatted at 2026-09-25 14:55:44 on dist-test-slave-v5vl"
server_key: "b0b2e3ebd03abe46e978084dfe7e4bca"
server_key_iv: "f517477017e6a15510bd0c4e847ce641"
server_key_version: "encryptionkey@0"
I20260925 14:56:30.326689 2252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:30.375644 2252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:30.379659 2252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:30.379967 2252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:30.381078 2252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:30.382433 2252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:30.382526 2252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:30.382609 2252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:30.382681 2252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:30.414594 2252 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:43323
I20260925 14:56:30.414705 2379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:43323 every 8 connection(s)
I20260925 14:56:30.415971 2252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:30.420507 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:30.418806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:33461)
0925 14:56:30.419494 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:30.419737 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
W20260925 14:56:30.421168 2380 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:34641 (0 consecutive failures): Network error: Failed to ping master at 127.17.133.254:34641: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
I20260925 14:56:30.421876 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:30.421605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:57687)
0925 14:56:30.421728 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:30.421787 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 14:56:30.422930 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:30.422569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:53365)
0925 14:56:30.422684 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:30.422781 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 14:56:30.423209 2380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:56:30.423218 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 2252
I20260925 14:56:30.423885 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:45953
--local_ip_for_outbound_sockets=127.17.133.195
--tserver_master_addrs=127.17.133.254:34641
--webserver_port=42373
--webserver_interface=127.17.133.195
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:30.740739 2383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:30.741008 2383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:30.741099 2383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:30.748138 2383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:30.748243 2383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:30.748353 2383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:56:30.758373 2383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:45953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=42373
--tserver_master_addrs=127.17.133.254:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:30.759663 2383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:30.760946 2383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:30.773731 2391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:30.773689 2389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:30.773664 2388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:30.774818 2383 server_base.cc:1034] running on GCE node
I20260925 14:56:30.775391 2383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:30.776288 2383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:30.777513 2383 hybrid_clock.cc:648] HybridClock initialized: now 1790348190777465 us; error 60 us; skew 500 ppm
I20260925 14:56:30.777837 2383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:30 dist-test-slave-v5vl krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348190, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:30.780900 2383 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:56:30.782318 2383 webserver.cc:460] Webserver started at http://127.17.133.195:42373/ using document root <none> and password file <none>
I20260925 14:56:30.782999 2383 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:30.783142 2383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:30.788726 2383 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 14:56:30.791213 2398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:30.792407 2383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:56:30.792569 2383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "a37d21a350aa4bd8837e75f19d6ea917"
format_stamp: "Formatted at 2026-09-25 14:55:45 on dist-test-slave-v5vl"
server_key: "a0606ff8447def512c36b0cb6aa435bd"
server_key_iv: "02f9cedccde45e13fcd1c50bad2c4fa8"
server_key_version: "encryptionkey@0"
I20260925 14:56:30.793695 2383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:30.806989 2383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:30.810422 2383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:30.810631 2383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:30.811401 2383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:30.812783 2383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:30.812868 2383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:30.812942 2383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:30.812995 2383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:30.845672 2383 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:45953
I20260925 14:56:30.845765 2510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:45953 every 8 connection(s)
I20260925 14:56:30.847244 2383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:30.852005 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:30.850267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:50931)
0925 14:56:30.850956 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:30.851194 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":263,"threads_started":1}
W20260925 14:56:30.852722 2511 heartbeater.cc:644] Failed to heartbeat to 127.17.133.254:34641 (0 consecutive failures): Network error: Failed to ping master at 127.17.133.254:34641: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
I20260925 14:56:30.853487 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:30.853199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:46343)
0925 14:56:30.853327 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:30.853389 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 14:56:30.854243 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:30.853954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:37841)
0925 14:56:30.854069 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:30.854139 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 14:56:30.854461 2511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:56:30.855270 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 2383
I20260925 14:56:30.950549 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:30.949789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:45789)
0925 14:56:30.950238 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:30.950333 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
I20260925 14:56:31.424688 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:31.423886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:33101)
0925 14:56:31.424323 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:31.424404 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260925 14:56:31.855773 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:31.855015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:60099)
0925 14:56:31.855447 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:31.855556 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":218,"threads_started":1}
I20260925 14:56:31.952247 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:31.951431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:49611)
0925 14:56:31.951964 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:31.952046 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":295,"threads_started":1}
I20260925 14:56:32.426149 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:32.425502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:42135)
0925 14:56:32.425920 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:32.425996 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":231,"threads_started":1}
I20260925 14:56:32.857394 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:32.856720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:44751)
0925 14:56:32.857165 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:32.857240 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":218,"threads_started":1}
I20260925 14:56:32.953948 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:32.953181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:35723)
0925 14:56:32.953656 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:32.953743 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
I20260925 14:56:33.427594 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:33.426941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:52665)
0925 14:56:33.427332 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:33.427423 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":221,"threads_started":1}
I20260925 14:56:33.859053 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:33.858310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:41019)
0925 14:56:33.858797 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:33.858878 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
I20260925 14:56:33.955487 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:33.954811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:48249)
0925 14:56:33.955212 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:33.955300 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":201,"threads_started":1}
I20260925 14:56:34.429237 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:34.428537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:34235)
0925 14:56:34.428996 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:34.429072 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":264,"threads_started":1}
I20260925 14:56:34.860808 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:34.860027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:43959)
0925 14:56:34.860465 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:34.860547 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260925 14:56:34.957149 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:34.956374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:37295)
0925 14:56:34.956895 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:34.956974 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":311,"threads_started":1}
I20260925 14:56:35.430826 2528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:35.430109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:38117)
0925 14:56:35.430573 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:35.430661 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":254,"threads_started":1}
I20260925 14:56:35.862380 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:35.861678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:57295)
0925 14:56:35.862132 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:35.862213 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":222,"threads_started":1}
I20260925 14:56:35.958681 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:35.957937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:58531)
0925 14:56:35.958415 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:35.958525 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":259,"threads_started":1}
I20260925 14:56:36.432333 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:36.431680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:33685)
0925 14:56:36.432096 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:36.432176 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260925 14:56:36.864105 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:36.863308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:35617)
0925 14:56:36.863838 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:36.863935 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":322,"threads_started":1}
I20260925 14:56:36.960309 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:36.959564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:44653)
0925 14:56:36.960019 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:36.960098 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":202,"threads_started":1}
I20260925 14:56:37.433949 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:37.433198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:59959)
0925 14:56:37.433658 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:37.433747 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":203,"threads_started":1}
I20260925 14:56:37.865839 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:37.865077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:41475)
0925 14:56:37.865546 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:37.865625 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":214,"threads_started":1}
I20260925 14:56:37.961944 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:37.961269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:50261)
0925 14:56:37.961691 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:37.961768 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":223,"threads_started":1}
I20260925 14:56:38.435618 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:38.434890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:42401)
0925 14:56:38.435374 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:38.435461 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
I20260925 14:56:38.867470 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:38.866792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:52363)
0925 14:56:38.867244 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:38.867317 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
I20260925 14:56:38.963620 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:38.962871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:54245)
0925 14:56:38.963344 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:38.963427 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":232,"threads_started":1}
I20260925 14:56:39.437165 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:39.436398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:55683)
0925 14:56:39.436882 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:39.436974 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":266,"threads_started":1}
I20260925 14:56:39.869140 2541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:39.868387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:54619)
0925 14:56:39.868900 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:39.868975 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
I20260925 14:56:39.965658 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:39.964855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:40507)
0925 14:56:39.965413 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:39.965495 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":330,"threads_started":1}
I20260925 14:56:40.438880 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:40.438066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:60661)
0925 14:56:40.438535 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:40.438672 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":243,"threads_started":1}
I20260925 14:56:40.870649 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:40.869951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:51915)
0925 14:56:40.870405 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:40.870484 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":240,"threads_started":1}
I20260925 14:56:40.967182 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:40.966503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:45045)
0925 14:56:40.966927 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:40.967007 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":203,"threads_started":1}
I20260925 14:56:41.440433 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:41.439739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:47485)
0925 14:56:41.440160 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:41.440253 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":216,"threads_started":1}
I20260925 14:56:41.872128 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:41.871486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:32769)
0925 14:56:41.871885 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:41.871963 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":205,"threads_started":1}
I20260925 14:56:41.968935 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:41.968218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:38501)
0925 14:56:41.968703 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:41.968781 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":244,"threads_started":1}
I20260925 14:56:42.441983 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:42.441304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:49733)
0925 14:56:42.441734 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:42.441827 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260925 14:56:42.873629 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:42.872992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:45629)
0925 14:56:42.873381 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:42.873454 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":203,"threads_started":1}
I20260925 14:56:42.970417 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:42.969756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:49159)
0925 14:56:42.970165 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:42.970249 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260925 14:56:43.443578 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:43.442878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:34463)
0925 14:56:43.443289 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:43.443392 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260925 14:56:43.875481 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:43.874721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:47555)
0925 14:56:43.875206 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:43.875297 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":246,"threads_started":1}
I20260925 14:56:43.972031 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:43.971307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:53647)
0925 14:56:43.971765 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:43.971847 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260925 14:56:44.445416 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:44.444459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:50523)
0925 14:56:44.445083 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:44.445185 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":293,"threads_started":1}
I20260925 14:56:44.877066 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:44.876324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:53695)
0925 14:56:44.876791 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:44.876878 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":217,"threads_started":1}
I20260925 14:56:44.973794 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:44.973110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.193:41069)
0925 14:56:44.973529 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:44.973620 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260925 14:56:45.447254 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:45.446555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.194:37389)
0925 14:56:45.446964 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:45.447085 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":208,"threads_started":1}
I20260925 14:56:45.866997 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 2080
I20260925 14:56:45.878806 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:45.878078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34641 (local address 127.17.133.195:51907)
0925 14:56:45.878538 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:45.878632 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.254:34641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":217,"threads_started":1}
I20260925 14:56:45.884848 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:45.868106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.17.133.193:38953)
0925 14:56:45.868804 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:45.868824 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:45.868871 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:56:45.870565 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:45.870574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:45.870993 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:45.871344 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:45.872814 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.874030 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.875358 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.876009 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.876207 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:45.879043 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:45.879053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:45.879078 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:45.879113 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:45.881724 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:45.882384 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:45.882391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:45.882397 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:45.882555 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:45.882910 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:45.882918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:45.882920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:45.883374 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:45.883679 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:45.884125 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 14:56:45.884532 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":346,"threads_started":1}
I20260925 14:56:45.886135 2244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:45.969447 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 2080
I20260925 14:56:45.986452 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 2252
I20260925 14:56:46.003680 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:45.987384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.17.133.194:43323)
0925 14:56:45.987862 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:45.987876 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:45.987913 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:56:45.989466 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:45.989477 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:45.989909 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:45.990274 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:45.991677 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.992964 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.994173 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.994853 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.995058 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:45.997549 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:45.997558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:45.997572 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:45.997608 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:46.000499 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:46.001243 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:46.001250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:46.001256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:46.001416 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:46.001792 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:46.001797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:46.001800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:46.002290 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:46.002588 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.003010 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.003401 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":224,"threads_started":1}
I20260925 14:56:46.005149 2375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:46.086884 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 2252
I20260925 14:56:46.104876 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 2383
I20260925 14:56:46.122826 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.105845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.17.133.195:45953)
0925 14:56:46.106460 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:46.106474 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:46.106515 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:56:46.108139 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:46.108149 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:46.108724 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:46.109110 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:46.110397 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.111608 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.112995 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.113676 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.113818 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.116314 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:46.116322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:46.116336 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:46.116371 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:46.119517 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:46.120248 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:46.120254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:46.120260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:46.120422 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:46.120834 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:46.120841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:46.120844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:46.121380 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:46.121665 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.122095 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.122497 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":289,"threads_started":1}
I20260925 14:56:46.124194 2506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:46.209251 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 2383
2026-09-25T14:56:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64300 ms)
[----------] 30 tests from SecurityITest (287946 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:56:46 dist-test-slave-v5vl krb5kdc[2568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:56:46 dist-test-slave-v5vl krb5kdc[2568](info): set up 2 sockets
Sep 25 14:56:46 dist-test-slave-v5vl krb5kdc[2568](info): commencing operation
krb5kdc: starting...
W20260925 14:56:47.300917 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:56:47 dist-test-slave-v5vl krb5kdc[2568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:56:47.540096 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34917
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:34917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:47.859215 2581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:47.859579 2581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:47.859750 2581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:47.866979 2581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:47.867120 2581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:47.867230 2581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:47.867336 2581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:47.867434 2581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:47.877187 2581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:34917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:34917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:47.878500 2581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:47.879781 2581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:47.891438 2586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:47.891381 2587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:47.891474 2589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:47.892257 2581 server_base.cc:1034] running on GCE node
I20260925 14:56:47.893162 2581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:47.894568 2581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:47.895829 2581 hybrid_clock.cc:648] HybridClock initialized: now 1790348207895750 us; error 58 us; skew 500 ppm
I20260925 14:56:47.896204 2581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:47 dist-test-slave-v5vl krb5kdc[2568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:47.902887 2581 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:56:47.904372 2581 webserver.cc:460] Webserver started at http://127.17.133.254:44977/ using document root <none> and password file <none>
I20260925 14:56:47.905150 2581 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:47.905305 2581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:47.905613 2581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:47.908510 2581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "a32c884658184863a1b091e168446586"
format_stamp: "Formatted at 2026-09-25 14:56:47 on dist-test-slave-v5vl"
server_key: "cc10fbea02dbbc8f9a09c06bd227f40d"
server_key_iv: "8200d62904d70a9ee75b107acfd58acc"
server_key_version: "encryptionkey@0"
I20260925 14:56:47.909255 2581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "a32c884658184863a1b091e168446586"
format_stamp: "Formatted at 2026-09-25 14:56:47 on dist-test-slave-v5vl"
server_key: "cc10fbea02dbbc8f9a09c06bd227f40d"
server_key_iv: "8200d62904d70a9ee75b107acfd58acc"
server_key_version: "encryptionkey@0"
I20260925 14:56:47.914050 2581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:56:47.917322 2596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:47.918596 2581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:56:47.918776 2581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "a32c884658184863a1b091e168446586"
format_stamp: "Formatted at 2026-09-25 14:56:47 on dist-test-slave-v5vl"
server_key: "cc10fbea02dbbc8f9a09c06bd227f40d"
server_key_iv: "8200d62904d70a9ee75b107acfd58acc"
server_key_version: "encryptionkey@0"
I20260925 14:56:47.918963 2581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:47.946316 2581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:47.950434 2581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:47.950753 2581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:47.968739 2581 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:34917
I20260925 14:56:47.968757 2647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:34917 every 8 connection(s)
I20260925 14:56:47.970031 2581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:47.973289 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 2581
I20260925 14:56:47.973534 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:56:47.973556 2648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:47.974186 17943 external_mini_cluster.cc:1428] Setting key e63ad1c028f196a5b023ea41f80dde27
I20260925 14:56:47.987844 2648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586: Bootstrap starting.
Sep 25 14:56:47 dist-test-slave-v5vl krb5kdc[2568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:47.991606 2648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:47.992764 2648 log.cc:824] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:47.995985 2648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586: No bootstrap required, opened a new log
I20260925 14:56:47.997084 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.976080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.17.133.254:34917)
0925 14:56:47.976903 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.976914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.976956 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:56:47.978466 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.978489 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.978855 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.979173 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:47.980964 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.982189 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.984093 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.984827 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.984960 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.989948 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:47.989956 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:47.989969 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:47.990009 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:47.993162 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.993893 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.993900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.993910 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.994119 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.994522 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.994525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.994527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.995017 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:47.995228 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.995691 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.995996 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":381,"threads_started":1}
I20260925 14:56:48.004256 2648 raft_consensus.cc:348] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32c884658184863a1b091e168446586" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34917 } }
I20260925 14:56:48.004666 2648 raft_consensus.cc:374] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:48.004774 2648 raft_consensus.cc:729] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32c884658184863a1b091e168446586, State: Initialized, Role: FOLLOWER
I20260925 14:56:48.005296 2648 consensus_queue.cc:260] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32c884658184863a1b091e168446586" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34917 } }
I20260925 14:56:48.005548 2648 raft_consensus.cc:388] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:48.005692 2648 raft_consensus.cc:482] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:48.005868 2648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:48.008165 2648 raft_consensus.cc:504] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32c884658184863a1b091e168446586" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34917 } }
I20260925 14:56:48.008719 2648 leader_election.cc:302] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a32c884658184863a1b091e168446586; no voters:
I20260925 14:56:48.009315 2648 leader_election.cc:288] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:56:48.009456 2653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:48.010154 2653 raft_consensus.cc:686] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [term 1 LEADER]: Becoming Leader. State: Replica: a32c884658184863a1b091e168446586, State: Running, Role: LEADER
I20260925 14:56:48.010665 2653 consensus_queue.cc:237] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32c884658184863a1b091e168446586" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34917 } }
I20260925 14:56:48.011333 2648 sys_catalog.cc:564] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:48.013499 2654 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a32c884658184863a1b091e168446586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32c884658184863a1b091e168446586" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34917 } } }
I20260925 14:56:48.014007 2654 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:48.014549 2655 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a32c884658184863a1b091e168446586. Latest consensus state: current_term: 1 leader_uuid: "a32c884658184863a1b091e168446586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32c884658184863a1b091e168446586" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 34917 } } }
I20260925 14:56:48.014890 2655 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586 [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:48.017616 2658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:48.023257 2658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:48.033139 2658 catalog_manager.cc:1347] Generated new cluster ID: a9966795f8e246d08ce96fedb2823aa2
I20260925 14:56:48.033284 2658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:48.068807 2658 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:48.070134 2658 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:48.077042 2658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586: Generated new TSK 0
I20260925 14:56:48.077883 2658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:56:48.152228 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34917
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:48.495700 2674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:48.495992 2674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:48.496088 2674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:48.503487 2674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:48.503647 2674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:48.503765 2674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:56:48.513383 2674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:48.514534 2674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:48.515755 2674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:48.528815 2680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:48.528919 2679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:48.528987 2682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:48.530002 2674 server_base.cc:1034] running on GCE node
I20260925 14:56:48.530689 2674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:48.531765 2674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:48.532999 2674 hybrid_clock.cc:648] HybridClock initialized: now 1790348208532942 us; error 55 us; skew 500 ppm
I20260925 14:56:48.533321 2674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:48 dist-test-slave-v5vl krb5kdc[2568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348208, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:48.536365 2674 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:56:48.537972 2674 webserver.cc:460] Webserver started at http://127.17.133.193:34753/ using document root <none> and password file <none>
I20260925 14:56:48.538666 2674 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:48.538815 2674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:48.539126 2674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:48.542057 2674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "4072adb6d20842e19a51410bb791892f"
format_stamp: "Formatted at 2026-09-25 14:56:48 on dist-test-slave-v5vl"
server_key: "35457eee406273873b579cd0ebec2d4a"
server_key_iv: "5bc9dddd1146e1eac2b2c797de2fc754"
server_key_version: "encryptionkey@0"
I20260925 14:56:48.542827 2674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "4072adb6d20842e19a51410bb791892f"
format_stamp: "Formatted at 2026-09-25 14:56:48 on dist-test-slave-v5vl"
server_key: "35457eee406273873b579cd0ebec2d4a"
server_key_iv: "5bc9dddd1146e1eac2b2c797de2fc754"
server_key_version: "encryptionkey@0"
I20260925 14:56:48.547773 2674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 14:56:48.551337 2689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:48.552670 2674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:48.552862 2674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "4072adb6d20842e19a51410bb791892f"
format_stamp: "Formatted at 2026-09-25 14:56:48 on dist-test-slave-v5vl"
server_key: "35457eee406273873b579cd0ebec2d4a"
server_key_iv: "5bc9dddd1146e1eac2b2c797de2fc754"
server_key_version: "encryptionkey@0"
I20260925 14:56:48.553061 2674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:48.582603 2674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:48.586838 2674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:48.587139 2674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:48.588086 2674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:48.589473 2674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:48.589581 2674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:48.589726 2674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:48.589805 2674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:48.622637 2674 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:34637
I20260925 14:56:48.622718 2801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:34637 every 8 connection(s)
I20260925 14:56:48.624066 2674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:48.629642 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 2674
I20260925 14:56:48.629854 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:56:48.630669 17943 external_mini_cluster.cc:1428] Setting key 1f6f54c46a4859ad117db6fac1c60760
Sep 25 14:56:48 dist-test-slave-v5vl krb5kdc[2568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348208, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:48.644546 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:48.626766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:57017 (local address 127.17.133.254:34917)
0925 14:56:48.627206 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:48.627211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:48.628192 (+ 981us) server_negotiation.cc:407] Connection header received
0925 14:56:48.630641 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:48.630646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:48.630718 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:48.630881 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:48.632647 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:48.633359 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:48.634887 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:48.635284 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:48.635415 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:48.639035 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:48.639040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:48.639043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:48.639073 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:48.641651 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:48.642369 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:48.642379 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:48.642381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:48.642493 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:48.642904 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:48.642908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:48.642911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:48.643161 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:48.643313 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:48.643991 (+ 678us) server_negotiation.cc:299] Negotiation successful
0925 14:56:48.644206 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260925 14:56:48.645534 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:48.627366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34917 (local address 127.17.133.193:57017)
0925 14:56:48.628038 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:48.628076 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:48.630309 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:48.631139 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:48.631147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:48.631784 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:48.632467 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:48.632488 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:48.633517 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:48.633523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:48.634738 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:48.634750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:48.635506 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:48.635514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:48.635752 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:48.636433 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:48.636457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:48.638817 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:48.641803 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:48.641811 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:48.641826 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:48.642241 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:48.642597 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:48.642603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:48.642607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:48.642788 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:48.643284 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:48.643292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:48.643614 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 14:56:48.643860 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 14:56:48.644363 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":274,"threads_started":1}
I20260925 14:56:48.647294 2802 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34917
I20260925 14:56:48.647612 2802 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:48.648305 2802 heartbeater.cc:511] Master 127.17.133.254:34917 requested a full tablet report, sending...
I20260925 14:56:48.650740 2613 ts_manager.cc:194] Registered new tserver with Master: 4072adb6d20842e19a51410bb791892f (127.17.133.193:34637)
I20260925 14:56:48.653069 2613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:57017
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:56:48.684185 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34917
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:49.007596 2808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:49.007872 2808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:49.007965 2808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:49.015049 2808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:49.015153 2808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:49.015265 2808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:56:49.025127 2808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:49.026332 2808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:49.027535 2808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:49.040303 2813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:49.040364 2816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:49.040385 2814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:49.041271 2808 server_base.cc:1034] running on GCE node
I20260925 14:56:49.041862 2808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:49.042827 2808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:49.044063 2808 hybrid_clock.cc:648] HybridClock initialized: now 1790348209044026 us; error 51 us; skew 500 ppm
I20260925 14:56:49.044381 2808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:49 dist-test-slave-v5vl krb5kdc[2568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348209, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:49.047508 2808 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:56:49.049182 2808 webserver.cc:460] Webserver started at http://127.17.133.194:44279/ using document root <none> and password file <none>
I20260925 14:56:49.049876 2808 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:49.050021 2808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:49.050344 2808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:49.053223 2808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "bc0fae976c5c49ab8fdd653a3e682cfa"
format_stamp: "Formatted at 2026-09-25 14:56:49 on dist-test-slave-v5vl"
server_key: "6314c46dcac55cd2442403579a058dd2"
server_key_iv: "846334ad24e6f3fda4d8be423ea671bd"
server_key_version: "encryptionkey@0"
I20260925 14:56:49.053965 2808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "bc0fae976c5c49ab8fdd653a3e682cfa"
format_stamp: "Formatted at 2026-09-25 14:56:49 on dist-test-slave-v5vl"
server_key: "6314c46dcac55cd2442403579a058dd2"
server_key_iv: "846334ad24e6f3fda4d8be423ea671bd"
server_key_version: "encryptionkey@0"
I20260925 14:56:49.059232 2808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:56:49.062556 2823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:49.063755 2808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:56:49.063925 2808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "bc0fae976c5c49ab8fdd653a3e682cfa"
format_stamp: "Formatted at 2026-09-25 14:56:49 on dist-test-slave-v5vl"
server_key: "6314c46dcac55cd2442403579a058dd2"
server_key_iv: "846334ad24e6f3fda4d8be423ea671bd"
server_key_version: "encryptionkey@0"
I20260925 14:56:49.064116 2808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:49.075747 2808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:49.079510 2808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:49.079785 2808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:49.080703 2808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:49.081926 2808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:49.082027 2808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:49.082185 2808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:49.082260 2808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:49.115340 2808 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:38291
I20260925 14:56:49.115384 2935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:38291 every 8 connection(s)
I20260925 14:56:49.116662 2808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:49.126175 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 2808
I20260925 14:56:49.126385 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:56:49.126816 17943 external_mini_cluster.cc:1428] Setting key 493eee47e0ef76f86e0e297db02fa7f8
Sep 25 14:56:49 dist-test-slave-v5vl krb5kdc[2568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348209, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:49.136498 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.119249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:50089 (local address 127.17.133.254:34917)
0925 14:56:49.119428 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:49.119433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:49.120610 (+ 1177us) server_negotiation.cc:407] Connection header received
0925 14:56:49.122513 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:49.122517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:49.122589 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:49.122676 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:49.124211 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.125058 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.126728 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.127207 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.127337 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.130767 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:49.130772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:49.130775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:49.130807 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:49.133491 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:49.134215 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:49.134221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:49.134225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:49.134359 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:49.134796 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:49.134802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:49.134806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:49.135069 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:49.135221 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:49.135904 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 14:56:49.136130 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:56:49.137508 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.119733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34917 (local address 127.17.133.194:50089)
0925 14:56:49.120435 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:49.120474 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:49.122220 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:49.122868 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:49.122875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:49.123462 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:49.124064 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.124081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.125190 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.125193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.126590 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.126603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.127432 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.127439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.127673 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.128357 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:49.128378 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:49.130537 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:49.133644 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:49.133653 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:49.133671 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:49.134090 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:49.134472 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:49.134479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:49.134482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:49.134686 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:49.135195 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:49.135203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:49.135552 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 14:56:49.135822 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 14:56:49.136364 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":294,"threads_started":1}
I20260925 14:56:49.138963 2936 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34917
I20260925 14:56:49.139289 2936 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:49.139997 2936 heartbeater.cc:511] Master 127.17.133.254:34917 requested a full tablet report, sending...
I20260925 14:56:49.141557 2613 ts_manager.cc:194] Registered new tserver with Master: bc0fae976c5c49ab8fdd653a3e682cfa (127.17.133.194:38291)
I20260925 14:56:49.142831 2613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:50089
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:56:49.179486 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34917
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:49.496961 2941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:49.497208 2941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:49.497298 2941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:49.504204 2941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:49.504300 2941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:49.504408 2941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:56:49.513777 2941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:34917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:49.514909 2941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:49.516078 2941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:49.529021 2947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:49.529122 2949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:49.529109 2946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:49.533947 2941 server_base.cc:1034] running on GCE node
I20260925 14:56:49.534529 2941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:49.535347 2941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:49.536561 2941 hybrid_clock.cc:648] HybridClock initialized: now 1790348209536534 us; error 46 us; skew 500 ppm
I20260925 14:56:49.536911 2941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:49 dist-test-slave-v5vl krb5kdc[2568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348209, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:49.539898 2941 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:56:49.541201 2941 webserver.cc:460] Webserver started at http://127.17.133.195:34539/ using document root <none> and password file <none>
I20260925 14:56:49.541829 2941 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:49.541922 2941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:49.542155 2941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:49.544806 2941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "3c70e308c6464fc690cde91d9c4f7332"
format_stamp: "Formatted at 2026-09-25 14:56:49 on dist-test-slave-v5vl"
server_key: "a95d23eb70c966dc72b029efa08943e4"
server_key_iv: "614e993f276bf24c43d5799012e1adf0"
server_key_version: "encryptionkey@0"
I20260925 14:56:49.545508 2941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "3c70e308c6464fc690cde91d9c4f7332"
format_stamp: "Formatted at 2026-09-25 14:56:49 on dist-test-slave-v5vl"
server_key: "a95d23eb70c966dc72b029efa08943e4"
server_key_iv: "614e993f276bf24c43d5799012e1adf0"
server_key_version: "encryptionkey@0"
I20260925 14:56:49.550575 2941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:56:49.553908 2956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:49.555228 2941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:56:49.555428 2941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "3c70e308c6464fc690cde91d9c4f7332"
format_stamp: "Formatted at 2026-09-25 14:56:49 on dist-test-slave-v5vl"
server_key: "a95d23eb70c966dc72b029efa08943e4"
server_key_iv: "614e993f276bf24c43d5799012e1adf0"
server_key_version: "encryptionkey@0"
I20260925 14:56:49.555683 2941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:49.572448 2941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:49.576807 2941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:49.577093 2941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:49.578022 2941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:49.579332 2941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:49.579447 2941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:49.579594 2941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:49.579680 2941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:49.612742 2941 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:42001
I20260925 14:56:49.612828 3068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:42001 every 8 connection(s)
I20260925 14:56:49.614037 2941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:49.621006 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 2941
I20260925 14:56:49.621212 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:56:49.621785 17943 external_mini_cluster.cc:1428] Setting key 837709c15ae34cf6589a03c58aa369ce
Sep 25 14:56:49 dist-test-slave-v5vl krb5kdc[2568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348209, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:49.635690 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.616627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:53529 (local address 127.17.133.254:34917)
0925 14:56:49.616818 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:49.616823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:49.617921 (+ 1098us) server_negotiation.cc:407] Connection header received
0925 14:56:49.619708 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:49.619713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:49.619782 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:49.619871 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:49.621710 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.622872 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.625292 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.625910 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.626028 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.629532 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:49.629537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:49.629540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:49.629574 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:49.632012 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:49.632806 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:49.632811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:49.632813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:49.633037 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:49.633593 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:49.633599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:49.633603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:49.633971 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:49.634135 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:49.635112 (+ 977us) server_negotiation.cc:299] Negotiation successful
0925 14:56:49.635336 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:56:49.636677 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.617057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:34917 (local address 127.17.133.195:53529)
0925 14:56:49.617735 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:49.617784 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:49.619433 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:49.620080 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:49.620116 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:49.620733 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:49.621541 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.621564 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.623145 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.623151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.625158 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.625169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.626108 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.626113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.626392 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.627117 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:49.627155 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:49.629325 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:49.632140 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:49.632146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:49.632157 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:49.632565 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:49.633185 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:49.633192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:49.633198 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:49.633457 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:49.634228 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:49.634238 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:49.634714 (+ 476us) client_negotiation.cc:769] Sending connection context
0925 14:56:49.635014 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 14:56:49.635510 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":269,"threads_started":1}
I20260925 14:56:49.638473 3069 heartbeater.cc:347] Connected to a master server at 127.17.133.254:34917
I20260925 14:56:49.638732 3069 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:49.639384 3069 heartbeater.cc:511] Master 127.17.133.254:34917 requested a full tablet report, sending...
I20260925 14:56:49.641216 2613 ts_manager.cc:194] Registered new tserver with Master: 3c70e308c6464fc690cde91d9c4f7332 (127.17.133.195:42001)
I20260925 14:56:49.642377 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:49.642369 2613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:53529
I20260925 14:56:49.656517 2802 heartbeater.cc:503] Master 127.17.133.254:34917 was elected leader, sending a full tablet report...
I20260925 14:56:49.659126 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.646807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.17.133.254:34917)
0925 14:56:49.646955 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:49.646959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:49.647389 (+ 430us) server_negotiation.cc:407] Connection header received
0925 14:56:49.647561 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:49.647564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:49.647634 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:49.647711 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:49.649120 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.649786 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.651120 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.651554 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.651647 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.653989 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:49.653994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:49.653997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:49.654025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:49.656419 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:49.657238 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:49.657243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:49.657246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:49.657396 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:49.657809 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:49.657814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:49.657817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:49.658199 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:49.658384 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:49.658731 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 14:56:49.658914 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:56:49.662710 2613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:49.669768 2613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:49.695598 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.681737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.17.133.195:42001)
0925 14:56:49.682709 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:49.682717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:49.683608 (+ 891us) server_negotiation.cc:407] Connection header received
0925 14:56:49.683947 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:49.683957 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:49.684271 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:49.684474 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:49.685710 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.687813 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.691895 (+ 4082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.693910 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.694117 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.694346 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:49.695014 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 14:56:49.695207 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":1162,"threads_started":1}
I20260925 14:56:49.696158 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.681097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:42001 (local address 127.0.0.1:44006)
0925 14:56:49.683484 (+ 2387us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:49.683501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:49.683737 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:49.684937 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:49.684948 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:49.684986 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:49.685468 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.685485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.688040 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.688048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.691663 (+ 3615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.691691 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.694529 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.694534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.694712 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.694789 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 14:56:49.695555 (+ 766us) client_negotiation.cc:240] Negotiation successful
0925 14:56:49.695782 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2261,"thread_start_us":914,"threads_started":1}
I20260925 14:56:49.700351 3004 tablet_service.cc:1467] Processing CreateTablet for tablet b8f2c23a77294039a00c5f4175e96871 (DEFAULT_TABLE table=test-table [id=53b27e66f57f47479dbd957e8ef2b41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:49.701483 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.681051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:38291 (local address 127.0.0.1:54154)
0925 14:56:49.681844 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:49.681880 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:49.682090 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:49.694801 (+ 12711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:49.694806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:49.694830 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:49.695246 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.695257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.697192 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.697198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.699206 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.699218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.700666 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.700672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.700819 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.700849 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 14:56:49.700950 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:56:49.701074 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":23,"thread_start_us":12530,"threads_started":1}
I20260925 14:56:49.701553 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.681120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.17.133.194:38291)
0925 14:56:49.694164 (+ 13044us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:49.694172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:49.694199 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:56:49.694300 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:49.694311 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:49.694586 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:49.694805 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:49.695462 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.697048 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.699364 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.700408 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.700610 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.700774 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:49.701002 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:56:49.701185 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12879,"thread_start_us":12913,"threads_started":1}
I20260925 14:56:49.701951 3004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f2c23a77294039a00c5f4175e96871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:49.702947 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.683155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.17.133.193:34637)
0925 14:56:49.694730 (+ 11575us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:49.694740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:49.694768 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:56:49.694861 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:49.694870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:49.695104 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:49.695295 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:49.696136 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.697650 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.700186 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.702045 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.702249 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.702400 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:49.702608 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 14:56:49.702721 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11392,"thread_start_us":11209,"threads_started":1}
I20260925 14:56:49.703042 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.682722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:34637 (local address 127.0.0.1:60242)
0925 14:56:49.693824 (+ 11102us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:49.693843 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:49.693968 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:49.695289 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:49.695297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:49.695320 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:49.695952 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.695966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.697818 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.697823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.700012 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.700026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.702244 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.702250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.702410 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.702439 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 14:56:49.702558 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 14:56:49.702692 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10958,"mutex_wait_us":1,"thread_start_us":651,"threads_started":1}
I20260925 14:56:49.704944 2871 tablet_service.cc:1467] Processing CreateTablet for tablet b8f2c23a77294039a00c5f4175e96871 (DEFAULT_TABLE table=test-table [id=53b27e66f57f47479dbd957e8ef2b41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:49.706010 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f2c23a77294039a00c5f4175e96871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:49.706485 2737 tablet_service.cc:1467] Processing CreateTablet for tablet b8f2c23a77294039a00c5f4175e96871 (DEFAULT_TABLE table=test-table [id=53b27e66f57f47479dbd957e8ef2b41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:49.707717 2737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f2c23a77294039a00c5f4175e96871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:49.716395 3087 tablet_bootstrap.cc:492] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: Bootstrap starting.
I20260925 14:56:49.718267 3088 tablet_bootstrap.cc:492] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: Bootstrap starting.
I20260925 14:56:49.720151 3087 tablet_bootstrap.cc:654] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:49.721526 3087 log.cc:824] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:49.721607 3089 tablet_bootstrap.cc:492] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: Bootstrap starting.
I20260925 14:56:49.721702 3088 tablet_bootstrap.cc:654] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:49.723012 3088 log.cc:824] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:49.725157 3087 tablet_bootstrap.cc:492] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: No bootstrap required, opened a new log
I20260925 14:56:49.725274 3089 tablet_bootstrap.cc:654] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:49.725421 3087 ts_tablet_manager.cc:1397] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 14:56:49.725929 3088 tablet_bootstrap.cc:492] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: No bootstrap required, opened a new log
I20260925 14:56:49.726186 3088 ts_tablet_manager.cc:1397] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 14:56:49.726366 3089 log.cc:824] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:49.728721 3089 tablet_bootstrap.cc:492] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: No bootstrap required, opened a new log
I20260925 14:56:49.728960 3089 ts_tablet_manager.cc:1397] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 14:56:49.735872 3087 raft_consensus.cc:348] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } }
I20260925 14:56:49.735991 3089 raft_consensus.cc:348] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } }
I20260925 14:56:49.736310 3087 raft_consensus.cc:374] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:49.736325 3089 raft_consensus.cc:374] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:49.736416 3089 raft_consensus.cc:729] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4072adb6d20842e19a51410bb791892f, State: Initialized, Role: FOLLOWER
I20260925 14:56:49.736413 3087 raft_consensus.cc:729] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc0fae976c5c49ab8fdd653a3e682cfa, State: Initialized, Role: FOLLOWER
I20260925 14:56:49.736552 3088 raft_consensus.cc:348] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } }
I20260925 14:56:49.736949 3088 raft_consensus.cc:374] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:49.737064 3088 raft_consensus.cc:729] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c70e308c6464fc690cde91d9c4f7332, State: Initialized, Role: FOLLOWER
I20260925 14:56:49.737059 3087 consensus_queue.cc:260] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } }
I20260925 14:56:49.737172 3089 consensus_queue.cc:260] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } }
I20260925 14:56:49.737584 3088 consensus_queue.cc:260] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } }
I20260925 14:56:49.738904 3089 ts_tablet_manager.cc:1428] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260925 14:56:49.744240 2936 heartbeater.cc:503] Master 127.17.133.254:34917 was elected leader, sending a full tablet report...
I20260925 14:56:49.745528 3087 ts_tablet_manager.cc:1428] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20260925 14:56:49.746464 3069 heartbeater.cc:503] Master 127.17.133.254:34917 was elected leader, sending a full tablet report...
I20260925 14:56:49.746909 3088 ts_tablet_manager.cc:1428] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: Time spent starting tablet: real 0.021s user 0.009s sys 0.012s
W20260925 14:56:49.865917 3070 tablet.cc:2369] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:56:49.868646 2937 tablet.cc:2369] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:56:49.876636 2803 tablet.cc:2369] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:49.914346 3093 raft_consensus.cc:482] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:49.914731 3093 raft_consensus.cc:504] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } }
I20260925 14:56:49.916918 3093 leader_election.cc:288] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c70e308c6464fc690cde91d9c4f7332 (127.17.133.195:42001), 4072adb6d20842e19a51410bb791892f (127.17.133.193:34637)
I20260925 14:56:49.923478 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.917151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:52037 (local address 127.17.133.195:42001)
0925 14:56:49.917430 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:49.917437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:49.917785 (+ 348us) server_negotiation.cc:407] Connection header received
0925 14:56:49.918057 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:49.918071 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:49.918172 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:49.918270 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:49.918895 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.919647 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.921591 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.922459 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.922672 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.922725 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:49.923049 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 14:56:49.923170 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:56:49.923789 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.917050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:42001 (local address 127.17.133.194:52037)
0925 14:56:49.917679 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:49.917695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:49.917877 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:49.918345 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:49.918349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:49.918376 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:49.918751 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.918759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.919784 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.919791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.921448 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.921460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.922697 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.922703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.922882 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.922919 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:56:49.923021 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:56:49.923163 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":347,"threads_started":1}
I20260925 14:56:49.924242 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.918149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:55991 (local address 127.17.133.193:34637)
0925 14:56:49.918359 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:49.918365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:49.918557 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:56:49.918762 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:49.918768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:49.918835 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:49.918934 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:49.919425 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.920325 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.922085 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.923075 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.923210 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.923289 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:49.923838 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 14:56:49.923965 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 14:56:49.924305 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.917823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:34637 (local address 127.17.133.194:55991)
0925 14:56:49.918461 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:49.918475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:49.918628 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:49.918948 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:49.918952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:49.918974 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:49.919278 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.919286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.920463 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.920468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.921935 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:49.921948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.923345 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.923353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:49.923551 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.923646 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 14:56:49.923813 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 14:56:49.923960 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":302,"threads_started":1}
I20260925 14:56:49.924991 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f2c23a77294039a00c5f4175e96871" candidate_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c70e308c6464fc690cde91d9c4f7332" is_pre_election: true
I20260925 14:56:49.925616 3024 raft_consensus.cc:2454] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc0fae976c5c49ab8fdd653a3e682cfa in term 0.
I20260925 14:56:49.925578 2757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f2c23a77294039a00c5f4175e96871" candidate_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4072adb6d20842e19a51410bb791892f" is_pre_election: true
I20260925 14:56:49.926117 2757 raft_consensus.cc:2454] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc0fae976c5c49ab8fdd653a3e682cfa in term 0.
I20260925 14:56:49.926383 2825 leader_election.cc:302] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c70e308c6464fc690cde91d9c4f7332, bc0fae976c5c49ab8fdd653a3e682cfa; no voters:
I20260925 14:56:49.926826 3093 raft_consensus.cc:2781] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:49.926993 3093 raft_consensus.cc:482] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:49.927076 3093 raft_consensus.cc:3037] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:49.929435 3093 raft_consensus.cc:504] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } }
I20260925 14:56:49.929945 3093 leader_election.cc:288] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [CANDIDATE]: Term 1 election: Requested vote from peers 3c70e308c6464fc690cde91d9c4f7332 (127.17.133.195:42001), 4072adb6d20842e19a51410bb791892f (127.17.133.193:34637)
I20260925 14:56:49.930401 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f2c23a77294039a00c5f4175e96871" candidate_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c70e308c6464fc690cde91d9c4f7332"
I20260925 14:56:49.930473 2757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f2c23a77294039a00c5f4175e96871" candidate_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4072adb6d20842e19a51410bb791892f"
I20260925 14:56:49.930595 3024 raft_consensus.cc:3037] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:49.930697 2757 raft_consensus.cc:3037] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:49.932880 3024 raft_consensus.cc:2454] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc0fae976c5c49ab8fdd653a3e682cfa in term 1.
I20260925 14:56:49.932927 2757 raft_consensus.cc:2454] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc0fae976c5c49ab8fdd653a3e682cfa in term 1.
I20260925 14:56:49.933257 2825 leader_election.cc:302] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c70e308c6464fc690cde91d9c4f7332, bc0fae976c5c49ab8fdd653a3e682cfa; no voters:
I20260925 14:56:49.933580 3093 raft_consensus.cc:2781] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:49.934002 3093 raft_consensus.cc:686] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 1 LEADER]: Becoming Leader. State: Replica: bc0fae976c5c49ab8fdd653a3e682cfa, State: Running, Role: LEADER
I20260925 14:56:49.934448 3093 consensus_queue.cc:237] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } }
I20260925 14:56:49.938717 2612 catalog_manager.cc:5495] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa reported cstate change: term changed from 0 to 1, leader changed from <none> to bc0fae976c5c49ab8fdd653a3e682cfa (127.17.133.194). New cstate: current_term: 1 leader_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0fae976c5c49ab8fdd653a3e682cfa" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 } health_report { overall_health: UNKNOWN } } }
I20260925 14:56:49.985419 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:49.979756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.17.133.194:38291)
0925 14:56:49.979989 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:49.979995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:49.980017 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:56:49.980145 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:49.980150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:49.980244 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:49.980384 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:49.980780 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.981914 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.983297 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:49.983813 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:49.983905 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:49.984352 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:49.984855 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:49.984962 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:49.985056 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:56:49.985118 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 14:56:49.993222 2757 raft_consensus.cc:1261] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [term 1 FOLLOWER]: Refusing update from remote peer bc0fae976c5c49ab8fdd653a3e682cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:49.993221 3024 raft_consensus.cc:1261] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [term 1 FOLLOWER]: Refusing update from remote peer bc0fae976c5c49ab8fdd653a3e682cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:49.994184 3098 consensus_queue.cc:1035] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4072adb6d20842e19a51410bb791892f" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:49.994529 3093 consensus_queue.cc:1035] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c70e308c6464fc690cde91d9c4f7332" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:50.021859 2613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:50.022138 2613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:50.025780 2613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a32c884658184863a1b091e168446586: Sending DeleteTablet for 3 replicas of tablet b8f2c23a77294039a00c5f4175e96871
I20260925 14:56:50.027104 2871 tablet_service.cc:1514] Processing DeleteTablet for tablet b8f2c23a77294039a00c5f4175e96871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:50 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:54154
I20260925 14:56:50.027149 3004 tablet_service.cc:1514] Processing DeleteTablet for tablet b8f2c23a77294039a00c5f4175e96871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:50 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:44006
I20260925 14:56:50.027486 2737 tablet_service.cc:1514] Processing DeleteTablet for tablet b8f2c23a77294039a00c5f4175e96871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:50 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:60242
I20260925 14:56:50.030267 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 2674
I20260925 14:56:50.038272 3113 tablet_replica.cc:330] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: stopping tablet replica
I20260925 14:56:50.038898 3113 raft_consensus.cc:2229] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:50.039553 3113 raft_consensus.cc:2258] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:50.042513 3113 ts_tablet_manager.cc:1905] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:50.042778 3112 tablet_replica.cc:330] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: stopping tablet replica
I20260925 14:56:50.043210 3112 raft_consensus.cc:2229] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:50.043684 3112 raft_consensus.cc:2258] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:50.044370 3111 tablet_replica.cc:330] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: stopping tablet replica
I20260925 14:56:50.044798 3111 raft_consensus.cc:2229] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:50.045526 3111 raft_consensus.cc:2258] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:50.045714 3112 ts_tablet_manager.cc:1905] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:50.047401 3111 ts_tablet_manager.cc:1905] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:50.052889 3112 ts_tablet_manager.cc:1918] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 25 14:56:50 dist-test-slave-v5vl krb5kdc[2568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:56:50.053048 3112 log.cc:1196] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/b8f2c23a77294039a00c5f4175e96871
I20260925 14:56:50.053570 3112 ts_tablet_manager.cc:1939] T b8f2c23a77294039a00c5f4175e96871 P 3c70e308c6464fc690cde91d9c4f7332: Deleting consensus metadata
I20260925 14:56:50.055382 2598 catalog_manager.cc:4841] TS 3c70e308c6464fc690cde91d9c4f7332 (127.17.133.195:42001): tablet b8f2c23a77294039a00c5f4175e96871 (table test-table [id=53b27e66f57f47479dbd957e8ef2b41d]) successfully deleted
I20260925 14:56:50.058068 3113 ts_tablet_manager.cc:1918] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:50.058247 3113 log.cc:1196] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/b8f2c23a77294039a00c5f4175e96871
I20260925 14:56:50.058764 3113 ts_tablet_manager.cc:1939] T b8f2c23a77294039a00c5f4175e96871 P 4072adb6d20842e19a51410bb791892f: Deleting consensus metadata
I20260925 14:56:50.060513 3111 ts_tablet_manager.cc:1918] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:50.060750 3111 log.cc:1196] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/b8f2c23a77294039a00c5f4175e96871
I20260925 14:56:50.061228 3111 ts_tablet_manager.cc:1939] T b8f2c23a77294039a00c5f4175e96871 P bc0fae976c5c49ab8fdd653a3e682cfa: Deleting consensus metadata
I20260925 14:56:50.062654 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:50.031956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.17.133.193:34637)
0925 14:56:50.032182 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:50.032191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:50.035738 (+ 3547us) server_negotiation.cc:407] Connection header received
0925 14:56:50.035930 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:50.035939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:50.036052 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:50.036171 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:50.044743 (+ 8572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:50.045846 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:50.048051 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:50.048665 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:50.048758 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:50.054655 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:50.054671 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:50.054687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:50.054728 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:50.059013 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:50.059910 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:50.059921 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:50.059925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:50.060107 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:50.060613 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:50.060619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:50.060622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:50.061092 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:50.061392 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:50.062000 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 14:56:50.062239 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:56:50.062745 2599 catalog_manager.cc:4841] TS bc0fae976c5c49ab8fdd653a3e682cfa (127.17.133.194:38291): tablet b8f2c23a77294039a00c5f4175e96871 (table test-table [id=53b27e66f57f47479dbd957e8ef2b41d]) successfully deleted
I20260925 14:56:50.064079 2797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:50.064035 2598 catalog_manager.cc:4841] TS 4072adb6d20842e19a51410bb791892f (127.17.133.193:34637): tablet b8f2c23a77294039a00c5f4175e96871 (table test-table [id=53b27e66f57f47479dbd957e8ef2b41d]) successfully deleted
I20260925 14:56:50.165709 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 2674
I20260925 14:56:50.185369 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 2808
Sep 25 14:56:50 dist-test-slave-v5vl krb5kdc[2568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:56:50.202775 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:50.186132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.17.133.194:38291)
0925 14:56:50.186299 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:50.186303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:50.186357 (+ 54us) server_negotiation.cc:407] Connection header received
0925 14:56:50.186534 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:50.186538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:50.186599 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:50.186688 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:50.188315 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:50.189210 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:50.191157 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:50.191580 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:50.191674 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:50.196832 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:50.196851 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:50.196873 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:50.196924 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:50.199932 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:50.200672 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:50.200683 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:50.200695 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:50.200846 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:50.201225 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:50.201230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:50.201235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:50.201579 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:50.201904 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:50.202180 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 14:56:50.202398 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:56:50.203459 2931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:50.315869 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 2808
I20260925 14:56:50.334685 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 2941
Sep 25 14:56:50 dist-test-slave-v5vl krb5kdc[2568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:56:50.349383 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:50.335462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.17.133.195:42001)
0925 14:56:50.335645 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:50.335652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:50.335696 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:56:50.335914 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:50.335924 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:50.335994 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:50.336100 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:50.337432 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:50.338141 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:50.339452 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:50.339834 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:50.339978 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:50.344199 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:50.344213 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:50.344228 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:50.344265 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:50.346780 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:50.347391 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:50.347398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:50.347400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:50.347539 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:50.347906 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:50.347909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:50.347911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:50.348235 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:50.348444 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:50.348989 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 14:56:50.349141 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:56:50.350059 3064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:50.439431 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 2941
I20260925 14:56:50.455660 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 2581
I20260925 14:56:50.456823 2643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:50.565271 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 2581
2026-09-25T14:56:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4358 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T14:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:50Z Disabled control of system clock
I20260925 14:56:50.637125 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37977
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:37977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:56:50.956126 3122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:50.956445 3122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:50.956573 3122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:50.963685 3122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:50.963797 3122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:50.963896 3122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:50.964010 3122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:50.964088 3122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:50.973210 3122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:37977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:50.974553 3122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:50.976385 3122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:50.987952 3130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:50.988036 3128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:50.988036 3127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:50.989195 3122 server_base.cc:1034] running on GCE node
I20260925 14:56:50.989840 3122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:50.991077 3122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:50.992270 3122 hybrid_clock.cc:648] HybridClock initialized: now 1790348210992239 us; error 26 us; skew 500 ppm
I20260925 14:56:50.992550 3122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:56:50.994823 3122 webserver.cc:460] Webserver started at http://127.17.133.254:35617/ using document root <none> and password file <none>
I20260925 14:56:50.995405 3122 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:50.995488 3122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:50.995730 3122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:50.998513 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "11de2f04401945dd94dc3e8371161c7b"
format_stamp: "Formatted at 2026-09-25 14:56:50 on dist-test-slave-v5vl"
server_key: "7c42675d412546f95a0ec37c58725713"
server_key_iv: "72581a7e5122480ffbf5b61ed7226648"
server_key_version: "encryptionkey@0"
I20260925 14:56:50.999228 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "11de2f04401945dd94dc3e8371161c7b"
format_stamp: "Formatted at 2026-09-25 14:56:50 on dist-test-slave-v5vl"
server_key: "7c42675d412546f95a0ec37c58725713"
server_key_iv: "72581a7e5122480ffbf5b61ed7226648"
server_key_version: "encryptionkey@0"
I20260925 14:56:51.003858 3122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 14:56:51.007131 3136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:51.008458 3122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:56:51.008689 3122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "11de2f04401945dd94dc3e8371161c7b"
format_stamp: "Formatted at 2026-09-25 14:56:50 on dist-test-slave-v5vl"
server_key: "7c42675d412546f95a0ec37c58725713"
server_key_iv: "72581a7e5122480ffbf5b61ed7226648"
server_key_version: "encryptionkey@0"
I20260925 14:56:51.008906 3122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:51.035714 3122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:51.036623 3122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:51.036917 3122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:51.054970 3122 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:37977
I20260925 14:56:51.054970 3187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:37977 every 8 connection(s)
I20260925 14:56:51.056345 3122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:51.060041 3188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:51.066756 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 3122
I20260925 14:56:51.066928 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:56:51.067346 17943 external_mini_cluster.cc:1428] Setting key 56684d776b0f6cd37024e95672587d39
I20260925 14:56:51.070683 3188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b: Bootstrap starting.
I20260925 14:56:51.073662 3188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:51.074632 3188 log.cc:824] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:51.077376 3188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b: No bootstrap required, opened a new log
I20260925 14:56:51.080837 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:51.069209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.17.133.254:37977)
0925 14:56:51.070134 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:51.070145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:51.070200 (+ 55us) server_negotiation.cc:407] Connection header received
0925 14:56:51.071892 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:51.071917 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:51.072358 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:51.072742 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:51.073267 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:51.074579 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:51.076394 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:51.077113 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:51.077305 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:51.078486 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:51.078494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:51.078508 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:51.079042 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:51.079130 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:51.079151 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:51.079284 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:51.079507 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 14:56:51.079816 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":318,"threads_started":1}
I20260925 14:56:51.085656 3188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11de2f04401945dd94dc3e8371161c7b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37977 } }
I20260925 14:56:51.086023 3188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:51.086131 3188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11de2f04401945dd94dc3e8371161c7b, State: Initialized, Role: FOLLOWER
I20260925 14:56:51.086724 3188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11de2f04401945dd94dc3e8371161c7b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37977 } }
I20260925 14:56:51.086961 3188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:51.087108 3188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:51.087287 3188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:51.089464 3188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11de2f04401945dd94dc3e8371161c7b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37977 } }
I20260925 14:56:51.089978 3188 leader_election.cc:302] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11de2f04401945dd94dc3e8371161c7b; no voters:
I20260925 14:56:51.090649 3188 leader_election.cc:288] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:56:51.090780 3193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:51.091637 3193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [term 1 LEADER]: Becoming Leader. State: Replica: 11de2f04401945dd94dc3e8371161c7b, State: Running, Role: LEADER
I20260925 14:56:51.092175 3193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11de2f04401945dd94dc3e8371161c7b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37977 } }
I20260925 14:56:51.093036 3188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:51.094911 3195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11de2f04401945dd94dc3e8371161c7b. Latest consensus state: current_term: 1 leader_uuid: "11de2f04401945dd94dc3e8371161c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11de2f04401945dd94dc3e8371161c7b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37977 } } }
I20260925 14:56:51.095278 3195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:51.095496 3194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11de2f04401945dd94dc3e8371161c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11de2f04401945dd94dc3e8371161c7b" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37977 } } }
I20260925 14:56:51.095872 3194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:51.097765 3198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:51.102587 3198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:51.111577 3198 catalog_manager.cc:1347] Generated new cluster ID: 2cf451468d9f4b1f9c47cd7b54b77d25
I20260925 14:56:51.111681 3198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:51.130493 3198 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:51.132308 3198 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:51.141326 3198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b: Generated new TSK 0
I20260925 14:56:51.142343 3198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:56:51.155759 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37977
--builtin_ntp_servers=127.17.133.212:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:56:51.484823 3212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:51.485097 3212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:51.485249 3212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:51.493026 3212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:51.493114 3212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:51.493314 3212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:56:51.502916 3212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:51.504408 3212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:51.506408 3212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:51.519887 3220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:51.519994 3218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:51.520010 3217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:51.521170 3212 server_base.cc:1034] running on GCE node
I20260925 14:56:51.521812 3212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:51.522759 3212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:51.524021 3212 hybrid_clock.cc:648] HybridClock initialized: now 1790348211523947 us; error 38 us; skew 500 ppm
I20260925 14:56:51.524353 3212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:56:51.531208 3212 webserver.cc:460] Webserver started at http://127.17.133.193:33259/ using document root <none> and password file <none>
I20260925 14:56:51.531917 3212 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:51.532050 3212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:51.532359 3212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:51.535388 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "0358f227ad814e31bc306e24cba7c004"
format_stamp: "Formatted at 2026-09-25 14:56:51 on dist-test-slave-v5vl"
server_key: "bd4f528c9e32f5e1c13eb99274cac32d"
server_key_iv: "de5434b38c82f3a07c9f7a412e02da4c"
server_key_version: "encryptionkey@0"
I20260925 14:56:51.536190 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "0358f227ad814e31bc306e24cba7c004"
format_stamp: "Formatted at 2026-09-25 14:56:51 on dist-test-slave-v5vl"
server_key: "bd4f528c9e32f5e1c13eb99274cac32d"
server_key_iv: "de5434b38c82f3a07c9f7a412e02da4c"
server_key_version: "encryptionkey@0"
I20260925 14:56:51.544445 3212 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260925 14:56:51.549324 3226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:51.551409 3212 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:56:51.551577 3212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "0358f227ad814e31bc306e24cba7c004"
format_stamp: "Formatted at 2026-09-25 14:56:51 on dist-test-slave-v5vl"
server_key: "bd4f528c9e32f5e1c13eb99274cac32d"
server_key_iv: "de5434b38c82f3a07c9f7a412e02da4c"
server_key_version: "encryptionkey@0"
I20260925 14:56:51.551801 3212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:51.567986 3212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:51.569016 3212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:51.569310 3212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:51.570758 3212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:51.572460 3212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:51.572574 3212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:51.572729 3212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:51.572817 3212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:51.610848 3212 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:40865
I20260925 14:56:51.610917 3338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:40865 every 8 connection(s)
I20260925 14:56:51.612284 3212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:51.613916 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 3212
I20260925 14:56:51.614097 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:56:51.614568 17943 external_mini_cluster.cc:1428] Setting key 976578a6b418dfcbeb1493b85ee0e907
I20260925 14:56:51.617327 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37977
--builtin_ntp_servers=127.17.133.212:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:56:51.625317 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:51.615002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:36711 (local address 127.17.133.254:37977)
0925 14:56:51.615460 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:51.615464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:51.616829 (+ 1365us) server_negotiation.cc:407] Connection header received
0925 14:56:51.618554 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:51.618563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:51.618654 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:51.618771 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:51.619745 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:51.620970 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:51.622490 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:51.623181 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:51.623318 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:51.624539 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:51.624548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:51.624556 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:51.624605 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:51.624658 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:51.624677 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:51.624773 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:51.625009 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 14:56:51.625082 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":250,"threads_started":1}
I20260925 14:56:51.626587 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:51.615637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37977 (local address 127.17.133.193:36711)
0925 14:56:51.616673 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:51.616716 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:51.618287 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:51.618948 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:51.618955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:51.619045 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:51.619580 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:51.619595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:51.621134 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:51.621137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:51.622341 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:51.622352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:51.623378 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:51.623382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:51.623566 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:51.624240 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:56:51.624242 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:51.624281 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:56:51.624352 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:51.624770 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:51.624778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:51.624784 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:56:51.624993 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 14:56:51.625398 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":320,"threads_started":1}
I20260925 14:56:51.628057 3339 heartbeater.cc:347] Connected to a master server at 127.17.133.254:37977
I20260925 14:56:51.628342 3339 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:51.629078 3339 heartbeater.cc:511] Master 127.17.133.254:37977 requested a full tablet report, sending...
I20260925 14:56:51.631079 3153 ts_manager.cc:194] Registered new tserver with Master: 0358f227ad814e31bc306e24cba7c004 (127.17.133.193:40865)
I20260925 14:56:51.632506 3153 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.193:36711
W20260925 14:56:51.942685 3343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:51.943003 3343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:51.943176 3343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:51.950634 3343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:51.950784 3343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:51.950992 3343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:56:51.960140 3343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:51.961486 3343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:51.963192 3343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:51.975656 3348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:51.975611 3349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:51.976121 3351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:51.976619 3343 server_base.cc:1034] running on GCE node
I20260925 14:56:51.977209 3343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:51.978108 3343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:51.979336 3343 hybrid_clock.cc:648] HybridClock initialized: now 1790348211979299 us; error 55 us; skew 500 ppm
I20260925 14:56:51.979662 3343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:56:51.981858 3343 webserver.cc:460] Webserver started at http://127.17.133.194:42017/ using document root <none> and password file <none>
I20260925 14:56:51.982512 3343 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:51.982633 3343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:51.982923 3343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:51.985723 3343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "828f61d5960b4171b6d4341b1c3ea94d"
format_stamp: "Formatted at 2026-09-25 14:56:51 on dist-test-slave-v5vl"
server_key: "e961069ad20fe08f4036d1dab545083b"
server_key_iv: "722be2107f72f246f123cf00ed08801a"
server_key_version: "encryptionkey@0"
I20260925 14:56:51.986502 3343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "828f61d5960b4171b6d4341b1c3ea94d"
format_stamp: "Formatted at 2026-09-25 14:56:51 on dist-test-slave-v5vl"
server_key: "e961069ad20fe08f4036d1dab545083b"
server_key_iv: "722be2107f72f246f123cf00ed08801a"
server_key_version: "encryptionkey@0"
I20260925 14:56:51.991156 3343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:56:51.994766 3357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:51.995913 3343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:56:51.996101 3343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "828f61d5960b4171b6d4341b1c3ea94d"
format_stamp: "Formatted at 2026-09-25 14:56:51 on dist-test-slave-v5vl"
server_key: "e961069ad20fe08f4036d1dab545083b"
server_key_iv: "722be2107f72f246f123cf00ed08801a"
server_key_version: "encryptionkey@0"
I20260925 14:56:51.996309 3343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:52.008358 3343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:52.009191 3343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:52.009496 3343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:52.010396 3343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:52.011816 3343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:52.011937 3343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:52.012074 3343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:52.012159 3343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:52.043680 3343 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:45575
I20260925 14:56:52.043712 3469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:45575 every 8 connection(s)
I20260925 14:56:52.045048 3343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:52.053030 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 3343
I20260925 14:56:52.053192 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:56:52.053619 17943 external_mini_cluster.cc:1428] Setting key c34b2cb0f825caa56a1cfbf09f6f2211
I20260925 14:56:52.056584 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37977
--builtin_ntp_servers=127.17.133.212:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:56:52.057792 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.047565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:41371 (local address 127.17.133.254:37977)
0925 14:56:52.047766 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.047772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.049154 (+ 1382us) server_negotiation.cc:407] Connection header received
0925 14:56:52.050837 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.050841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.050904 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.050989 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:52.051929 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.052658 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.054502 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.055203 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.055325 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.056916 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:52.056925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:52.056932 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:52.056955 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:52.056991 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:52.057010 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:52.057072 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.057387 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.057482 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:56:52.059136 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.048107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37977 (local address 127.17.133.194:41371)
0925 14:56:52.049002 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:52.049042 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:52.050601 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:52.051187 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:52.051194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:52.051270 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:52.051802 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.051818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.052864 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.052869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.054292 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.054307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.055418 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.055426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.055716 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.056533 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:56:52.056536 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:52.056619 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:56:52.056714 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:52.057164 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:52.057177 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:52.057188 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:56:52.057354 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 14:56:52.057820 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":248,"threads_started":1}
I20260925 14:56:52.060539 3470 heartbeater.cc:347] Connected to a master server at 127.17.133.254:37977
I20260925 14:56:52.060835 3470 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:52.061616 3470 heartbeater.cc:511] Master 127.17.133.254:37977 requested a full tablet report, sending...
I20260925 14:56:52.063136 3153 ts_manager.cc:194] Registered new tserver with Master: 828f61d5960b4171b6d4341b1c3ea94d (127.17.133.194:45575)
I20260925 14:56:52.064353 3153 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.194:41371
W20260925 14:56:52.386498 3473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:52.386759 3473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:52.386917 3473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:52.394263 3473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:52.394349 3473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:52.394516 3473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:56:52.403618 3473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:52.405100 3473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:52.406809 3473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:52.420022 3478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:52.419977 3481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:52.420048 3479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:52.420922 3473 server_base.cc:1034] running on GCE node
I20260925 14:56:52.421598 3473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:52.422607 3473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:52.423820 3473 hybrid_clock.cc:648] HybridClock initialized: now 1790348212423765 us; error 41 us; skew 500 ppm
I20260925 14:56:52.424157 3473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:56:52.426412 3473 webserver.cc:460] Webserver started at http://127.17.133.195:38515/ using document root <none> and password file <none>
I20260925 14:56:52.427098 3473 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:52.427215 3473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:52.427524 3473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:52.430330 3473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "325a2a7497b346d08b46c42fe7fe3b14"
format_stamp: "Formatted at 2026-09-25 14:56:52 on dist-test-slave-v5vl"
server_key: "18667978d47e2893ba04591bfe5c3f24"
server_key_iv: "97fb1275f4fe91d60a235251d809b53b"
server_key_version: "encryptionkey@0"
I20260925 14:56:52.431130 3473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "325a2a7497b346d08b46c42fe7fe3b14"
format_stamp: "Formatted at 2026-09-25 14:56:52 on dist-test-slave-v5vl"
server_key: "18667978d47e2893ba04591bfe5c3f24"
server_key_iv: "97fb1275f4fe91d60a235251d809b53b"
server_key_version: "encryptionkey@0"
I20260925 14:56:52.436201 3473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 14:56:52.439551 3487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:52.441038 3473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:52.441208 3473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "325a2a7497b346d08b46c42fe7fe3b14"
format_stamp: "Formatted at 2026-09-25 14:56:52 on dist-test-slave-v5vl"
server_key: "18667978d47e2893ba04591bfe5c3f24"
server_key_iv: "97fb1275f4fe91d60a235251d809b53b"
server_key_version: "encryptionkey@0"
I20260925 14:56:52.441414 3473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:52.452687 3473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:52.453364 3473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:52.453657 3473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:52.454526 3473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:52.455824 3473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:52.455940 3473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:52.456056 3473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:52.456116 3473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:52.488533 3473 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:39131
I20260925 14:56:52.488647 3599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:39131 every 8 connection(s)
I20260925 14:56:52.489904 3473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:52.499975 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 3473
I20260925 14:56:52.500154 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:56:52.500742 17943 external_mini_cluster.cc:1428] Setting key 324c5352fe5402b9902e7331d476150e
I20260925 14:56:52.502192 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.492250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:57037 (local address 127.17.133.254:37977)
0925 14:56:52.492426 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.492431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.493838 (+ 1407us) server_negotiation.cc:407] Connection header received
0925 14:56:52.495657 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.495662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.495764 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.495866 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:52.496959 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.497690 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.499224 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.499700 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.499833 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.501418 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:52.501423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:52.501426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:52.501444 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:52.501476 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:52.501487 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:52.501582 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.501800 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.501893 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:56:52.503057 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.492747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37977 (local address 127.17.133.195:57037)
0925 14:56:52.493685 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:52.493731 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:52.495397 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:52.496101 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:52.496109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:52.496191 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:52.496790 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.496808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.497850 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.497855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.499078 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.499089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.499930 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.499937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.500282 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.501112 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:56:52.501114 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:52.501153 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:56:52.501218 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:52.501581 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:52.501593 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:52.501602 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 14:56:52.501778 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 14:56:52.502130 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":296,"threads_started":1}
I20260925 14:56:52.504278 3600 heartbeater.cc:347] Connected to a master server at 127.17.133.254:37977
I20260925 14:56:52.504552 3600 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:52.505249 3600 heartbeater.cc:511] Master 127.17.133.254:37977 requested a full tablet report, sending...
I20260925 14:56:52.506528 3153 ts_manager.cc:194] Registered new tserver with Master: 325a2a7497b346d08b46c42fe7fe3b14 (127.17.133.195:39131)
I20260925 14:56:52.507292 3153 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.195:57037
I20260925 14:56:52.515328 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:52.524668 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.519229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.17.133.254:37977)
0925 14:56:52.519485 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.519493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.519805 (+ 312us) server_negotiation.cc:407] Connection header received
0925 14:56:52.520042 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.520046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.520117 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.520198 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:52.520728 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.521369 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.522619 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.522977 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.523060 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.524023 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:52.524027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:52.524030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:52.524044 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:52.524070 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:52.524080 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:52.524140 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.524255 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.524314 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:56:52.527858 3153 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:52.534773 3153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:52.558270 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.547195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.17.133.194:45575)
0925 14:56:52.550952 (+ 3757us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.550959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.550984 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:56:52.551084 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.551092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.551320 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.551507 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:52.552149 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.553574 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.555617 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.557055 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.557243 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.557465 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.557726 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.557887 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3597,"thread_start_us":3467,"threads_started":1}
I20260925 14:56:52.558322 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.547342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:39131 (local address 127.0.0.1:60306)
0925 14:56:52.549057 (+ 1715us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:52.549071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:52.549159 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:52.550335 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:52.550343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:52.550379 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:52.550724 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.550740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.553253 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.553259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.555673 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.555689 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.557396 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.557406 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.557573 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.557610 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:56:52.557664 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:56:52.557878 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1609,"mutex_wait_us":4,"thread_start_us":1498,"threads_started":1}
I20260925 14:56:52.558351 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.546555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:45575 (local address 127.0.0.1:49084)
0925 14:56:52.547743 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:52.547791 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:52.547989 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:52.551512 (+ 3523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:52.551517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:52.551540 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:52.551962 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.551976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.553709 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.553714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.555382 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.555395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.557250 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.557256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.557461 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.557523 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 14:56:52.557654 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 14:56:52.557883 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":69,"thread_start_us":743,"threads_started":1}
I20260925 14:56:52.558882 3613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.546763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:40865 (local address 127.0.0.1:34812)
0925 14:56:52.548475 (+ 1712us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:52.548495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:52.548627 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:52.551829 (+ 3202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:52.551834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:52.551860 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:52.552348 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.552359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.553968 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.553974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.556178 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.556191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.558038 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.558046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.558255 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.558320 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 14:56:52.558414 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:56:52.558539 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1547,"mutex_wait_us":38,"thread_start_us":1140,"threads_started":1}
I20260925 14:56:52.559448 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.547780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.17.133.195:39131)
0925 14:56:52.549848 (+ 2068us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.549854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.549898 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:56:52.549967 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.549973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.550148 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.550328 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:52.550940 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.553119 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.555899 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.557100 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.558647 (+ 1547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.558848 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.558919 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.559114 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":1795,"threads_started":1}
I20260925 14:56:52.559859 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.547059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.17.133.193:40865)
0925 14:56:52.551264 (+ 4205us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.551270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.551291 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:56:52.551367 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.551371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.551570 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.551810 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:52.552541 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.553812 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.556381 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.557830 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.559150 (+ 1320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.559282 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.559335 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.559499 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4076,"thread_start_us":3949,"threads_started":1}
I20260925 14:56:52.562891 3405 tablet_service.cc:1467] Processing CreateTablet for tablet 27ab572901ab4752af6c6bfaddd1da80 (DEFAULT_TABLE table=test-table [id=4992c6bfa0894de295ae6a15b80237c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:52.563241 3274 tablet_service.cc:1467] Processing CreateTablet for tablet 27ab572901ab4752af6c6bfaddd1da80 (DEFAULT_TABLE table=test-table [id=4992c6bfa0894de295ae6a15b80237c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:52.563241 3535 tablet_service.cc:1467] Processing CreateTablet for tablet 27ab572901ab4752af6c6bfaddd1da80 (DEFAULT_TABLE table=test-table [id=4992c6bfa0894de295ae6a15b80237c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:52.564090 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ab572901ab4752af6c6bfaddd1da80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:52.564512 3274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ab572901ab4752af6c6bfaddd1da80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:52.565258 3535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ab572901ab4752af6c6bfaddd1da80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:52.575209 3618 tablet_bootstrap.cc:492] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: Bootstrap starting.
I20260925 14:56:52.578711 3618 tablet_bootstrap.cc:654] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:52.579707 3618 log.cc:824] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:52.581655 3620 tablet_bootstrap.cc:492] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: Bootstrap starting.
I20260925 14:56:52.582556 3618 tablet_bootstrap.cc:492] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: No bootstrap required, opened a new log
I20260925 14:56:52.582826 3618 ts_tablet_manager.cc:1397] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 14:56:52.586126 3620 tablet_bootstrap.cc:654] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:52.587257 3619 tablet_bootstrap.cc:492] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: Bootstrap starting.
I20260925 14:56:52.587410 3620 log.cc:824] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:52.590085 3620 tablet_bootstrap.cc:492] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: No bootstrap required, opened a new log
I20260925 14:56:52.590370 3620 ts_tablet_manager.cc:1397] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260925 14:56:52.590780 3619 tablet_bootstrap.cc:654] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:52.591997 3619 log.cc:824] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:52.592291 3618 raft_consensus.cc:348] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } }
I20260925 14:56:52.592842 3618 raft_consensus.cc:374] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:52.592988 3618 raft_consensus.cc:729] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0358f227ad814e31bc306e24cba7c004, State: Initialized, Role: FOLLOWER
I20260925 14:56:52.593780 3618 consensus_queue.cc:260] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } }
I20260925 14:56:52.594873 3339 heartbeater.cc:503] Master 127.17.133.254:37977 was elected leader, sending a full tablet report...
I20260925 14:56:52.595256 3619 tablet_bootstrap.cc:492] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: No bootstrap required, opened a new log
I20260925 14:56:52.595515 3619 ts_tablet_manager.cc:1397] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260925 14:56:52.595569 3618 ts_tablet_manager.cc:1428] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 14:56:52.599200 3620 raft_consensus.cc:348] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } }
I20260925 14:56:52.599611 3620 raft_consensus.cc:374] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:52.599709 3620 raft_consensus.cc:729] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325a2a7497b346d08b46c42fe7fe3b14, State: Initialized, Role: FOLLOWER
I20260925 14:56:52.600428 3620 consensus_queue.cc:260] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } }
I20260925 14:56:52.601791 3600 heartbeater.cc:503] Master 127.17.133.254:37977 was elected leader, sending a full tablet report...
I20260925 14:56:52.602471 3620 ts_tablet_manager.cc:1428] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 14:56:52.604449 3619 raft_consensus.cc:348] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } }
I20260925 14:56:52.604905 3619 raft_consensus.cc:374] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:52.605011 3619 raft_consensus.cc:729] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828f61d5960b4171b6d4341b1c3ea94d, State: Initialized, Role: FOLLOWER
I20260925 14:56:52.605564 3619 consensus_queue.cc:260] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } }
I20260925 14:56:52.606630 3470 heartbeater.cc:503] Master 127.17.133.254:37977 was elected leader, sending a full tablet report...
I20260925 14:56:52.607126 3619 ts_tablet_manager.cc:1428] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260925 14:56:52.614663 3340 tablet.cc:2369] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:52.621532 3625 raft_consensus.cc:482] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:52.621809 3625 raft_consensus.cc:504] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } }
I20260925 14:56:52.623394 3625 leader_election.cc:288] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 828f61d5960b4171b6d4341b1c3ea94d (127.17.133.194:45575), 0358f227ad814e31bc306e24cba7c004 (127.17.133.193:40865)
I20260925 14:56:52.629500 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.623573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:45575 (local address 127.17.133.195:52575)
0925 14:56:52.623822 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:52.623844 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:52.623976 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:52.624419 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:52.624425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:52.624463 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:52.624752 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.624761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.625741 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.625748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.627656 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.627669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.628754 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.628760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.629000 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.629045 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:56:52.629138 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:56:52.629270 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":26,"spinlock_wait_cycles":20224}
I20260925 14:56:52.629563 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.623637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:52575 (local address 127.17.133.194:45575)
0925 14:56:52.624059 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.624067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.624094 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:56:52.624183 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.624188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.624267 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.624367 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:52.624899 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.625587 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.627808 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.628553 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.628737 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.628788 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.629171 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.629318 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260925 14:56:52.630541 3425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab572901ab4752af6c6bfaddd1da80" candidate_uuid: "325a2a7497b346d08b46c42fe7fe3b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "828f61d5960b4171b6d4341b1c3ea94d" is_pre_election: true
I20260925 14:56:52.631043 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.623786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:40865 (local address 127.17.133.195:40269)
0925 14:56:52.624419 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:52.624435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:52.624555 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:52.625769 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:52.625773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:52.625799 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:52.626218 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.626231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.627347 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.627352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.628969 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:52.628981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.630466 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.630470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:52.630579 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.630608 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 14:56:52.630674 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:56:52.630753 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":20,"thread_start_us":364,"threads_started":1}
I20260925 14:56:52.631095 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.623788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:40269 (local address 127.17.133.193:40865)
0925 14:56:52.623970 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.623976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.624537 (+ 561us) server_negotiation.cc:407] Connection header received
0925 14:56:52.624697 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.624701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.624768 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.624839 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:52.626391 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.627207 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.629172 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.630319 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.630476 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.630555 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.630737 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.630866 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:56:52.631140 3425 raft_consensus.cc:2454] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 325a2a7497b346d08b46c42fe7fe3b14 in term 0.
I20260925 14:56:52.631893 3490 leader_election.cc:302] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 325a2a7497b346d08b46c42fe7fe3b14, 828f61d5960b4171b6d4341b1c3ea94d; no voters:
I20260925 14:56:52.632324 3625 raft_consensus.cc:2781] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:52.632213 3294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab572901ab4752af6c6bfaddd1da80" candidate_uuid: "325a2a7497b346d08b46c42fe7fe3b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0358f227ad814e31bc306e24cba7c004" is_pre_election: true
I20260925 14:56:52.632472 3625 raft_consensus.cc:482] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:52.632614 3625 raft_consensus.cc:3037] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:52.632719 3294 raft_consensus.cc:2454] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 325a2a7497b346d08b46c42fe7fe3b14 in term 0.
I20260925 14:56:52.634865 3625 raft_consensus.cc:504] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } }
I20260925 14:56:52.635423 3625 leader_election.cc:288] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [CANDIDATE]: Term 1 election: Requested vote from peers 828f61d5960b4171b6d4341b1c3ea94d (127.17.133.194:45575), 0358f227ad814e31bc306e24cba7c004 (127.17.133.193:40865)
I20260925 14:56:52.635828 3425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab572901ab4752af6c6bfaddd1da80" candidate_uuid: "325a2a7497b346d08b46c42fe7fe3b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "828f61d5960b4171b6d4341b1c3ea94d"
I20260925 14:56:52.635890 3294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab572901ab4752af6c6bfaddd1da80" candidate_uuid: "325a2a7497b346d08b46c42fe7fe3b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0358f227ad814e31bc306e24cba7c004"
I20260925 14:56:52.636041 3425 raft_consensus.cc:3037] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:52.636106 3294 raft_consensus.cc:3037] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:52.638293 3294 raft_consensus.cc:2454] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 325a2a7497b346d08b46c42fe7fe3b14 in term 1.
I20260925 14:56:52.638293 3425 raft_consensus.cc:2454] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 325a2a7497b346d08b46c42fe7fe3b14 in term 1.
I20260925 14:56:52.638710 3490 leader_election.cc:302] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0358f227ad814e31bc306e24cba7c004, 325a2a7497b346d08b46c42fe7fe3b14, 828f61d5960b4171b6d4341b1c3ea94d; no voters:
I20260925 14:56:52.639024 3625 raft_consensus.cc:2781] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:52.639436 3625 raft_consensus.cc:686] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 1 LEADER]: Becoming Leader. State: Replica: 325a2a7497b346d08b46c42fe7fe3b14, State: Running, Role: LEADER
I20260925 14:56:52.640020 3625 consensus_queue.cc:237] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } }
I20260925 14:56:52.644510 3152 catalog_manager.cc:5495] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 325a2a7497b346d08b46c42fe7fe3b14 (127.17.133.195). New cstate: current_term: 1 leader_uuid: "325a2a7497b346d08b46c42fe7fe3b14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "325a2a7497b346d08b46c42fe7fe3b14" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 39131 } health_report { overall_health: HEALTHY } } }
I20260925 14:56:52.682454 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.677020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.17.133.195:39131)
0925 14:56:52.677233 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.677239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.677343 (+ 104us) server_negotiation.cc:407] Connection header received
0925 14:56:52.677590 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.677594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.677680 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.677837 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:52.678190 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.678913 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.680393 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.680826 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.680961 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.681422 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:52.681916 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:52.682073 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.682131 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.682207 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 14:56:52.690167 3294 raft_consensus.cc:1261] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [term 1 FOLLOWER]: Refusing update from remote peer 325a2a7497b346d08b46c42fe7fe3b14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:52.690181 3425 raft_consensus.cc:1261] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [term 1 FOLLOWER]: Refusing update from remote peer 325a2a7497b346d08b46c42fe7fe3b14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:52.691136 3628 consensus_queue.cc:1035] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "828f61d5960b4171b6d4341b1c3ea94d" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 45575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:52.691453 3625 consensus_queue.cc:1035] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0358f227ad814e31bc306e24cba7c004" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 40865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:52.726177 3152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:52.726487 3152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:52.730031 3152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11de2f04401945dd94dc3e8371161c7b: Sending DeleteTablet for 3 replicas of tablet 27ab572901ab4752af6c6bfaddd1da80
I20260925 14:56:52.731081 3405 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ab572901ab4752af6c6bfaddd1da80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:52 UTC) from {username='slave'} at 127.0.0.1:49084
I20260925 14:56:52.731266 3535 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ab572901ab4752af6c6bfaddd1da80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:52 UTC) from {username='slave'} at 127.0.0.1:60306
I20260925 14:56:52.731310 3274 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ab572901ab4752af6c6bfaddd1da80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:52 UTC) from {username='slave'} at 127.0.0.1:34812
I20260925 14:56:52.735680 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 3212
I20260925 14:56:52.743605 3642 tablet_replica.cc:330] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: stopping tablet replica
I20260925 14:56:52.744053 3642 raft_consensus.cc:2229] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:52.744620 3642 raft_consensus.cc:2258] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:52.746887 3642 ts_tablet_manager.cc:1905] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:52.753978 3642 ts_tablet_manager.cc:1918] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:52.754148 3642 log.cc:1196] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/27ab572901ab4752af6c6bfaddd1da80
I20260925 14:56:52.754731 3642 ts_tablet_manager.cc:1939] T 27ab572901ab4752af6c6bfaddd1da80 P 0358f227ad814e31bc306e24cba7c004: Deleting consensus metadata
I20260925 14:56:52.756714 3641 tablet_replica.cc:330] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: stopping tablet replica
I20260925 14:56:52.757187 3641 raft_consensus.cc:2229] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:52.757902 3640 tablet_replica.cc:330] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: stopping tablet replica
I20260925 14:56:52.757984 3641 raft_consensus.cc:2258] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:52.758409 3640 raft_consensus.cc:2229] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:52.758900 3640 raft_consensus.cc:2258] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:52.758875 3137 catalog_manager.cc:4841] TS 0358f227ad814e31bc306e24cba7c004 (127.17.133.193:40865): tablet 27ab572901ab4752af6c6bfaddd1da80 (table test-table [id=4992c6bfa0894de295ae6a15b80237c5]) successfully deleted
I20260925 14:56:52.760844 3641 ts_tablet_manager.cc:1905] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:52.761533 3640 ts_tablet_manager.cc:1905] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:52.767063 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.737072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.17.133.193:40865)
0925 14:56:52.737301 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.737310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.757193 (+ 19883us) server_negotiation.cc:407] Connection header received
0925 14:56:52.757438 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.757446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.758877 (+ 1431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.759007 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:52.760015 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.761152 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.763057 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.763706 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.763799 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.765326 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:52.765337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:52.765355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:52.766148 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:52.766207 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:52.766231 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:52.766319 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.766571 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.766695 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 14:56:52.768333 3334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:52.768863 3640 ts_tablet_manager.cc:1918] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:52.769040 3640 log.cc:1196] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/27ab572901ab4752af6c6bfaddd1da80
I20260925 14:56:52.769834 3640 ts_tablet_manager.cc:1939] T 27ab572901ab4752af6c6bfaddd1da80 P 828f61d5960b4171b6d4341b1c3ea94d: Deleting consensus metadata
I20260925 14:56:52.772377 3139 catalog_manager.cc:4841] TS 828f61d5960b4171b6d4341b1c3ea94d (127.17.133.194:45575): tablet 27ab572901ab4752af6c6bfaddd1da80 (table test-table [id=4992c6bfa0894de295ae6a15b80237c5]) successfully deleted
I20260925 14:56:52.772924 3641 ts_tablet_manager.cc:1918] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:52.773130 3641 log.cc:1196] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/27ab572901ab4752af6c6bfaddd1da80
I20260925 14:56:52.773710 3641 ts_tablet_manager.cc:1939] T 27ab572901ab4752af6c6bfaddd1da80 P 325a2a7497b346d08b46c42fe7fe3b14: Deleting consensus metadata
I20260925 14:56:52.775496 3138 catalog_manager.cc:4841] TS 325a2a7497b346d08b46c42fe7fe3b14 (127.17.133.195:39131): tablet 27ab572901ab4752af6c6bfaddd1da80 (table test-table [id=4992c6bfa0894de295ae6a15b80237c5]) successfully deleted
I20260925 14:56:52.876247 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 3212
I20260925 14:56:52.893844 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 3343
I20260925 14:56:52.900717 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:52.894647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.17.133.194:45575)
0925 14:56:52.894815 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:52.894823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:52.894844 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:56:52.895058 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:52.895062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:52.895158 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:52.895268 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:52.895647 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.896333 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.897676 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:52.898062 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:52.898190 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:52.899336 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:52.899345 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:52.899359 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:52.900005 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:52.900052 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:52.900071 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:52.900206 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:52.900284 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 14:56:52.900381 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:56:52.901538 3465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:52.989717 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 3343
I20260925 14:56:53.006695 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 3473
I20260925 14:56:53.018354 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:53.007482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.17.133.195:39131)
0925 14:56:53.007676 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:53.007682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:53.007704 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:56:53.007779 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:53.007784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:53.007885 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:53.008007 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:53.008372 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:53.009620 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:53.010876 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:53.011593 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:53.011730 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:53.016641 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:53.016653 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:53.016671 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:53.017529 (+ 858us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:53.017600 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:53.017628 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:53.017750 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:53.017873 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 14:56:53.017968 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 14:56:53.019086 3595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:53.153412 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 3473
I20260925 14:56:53.172709 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 3122
I20260925 14:56:53.173743 3183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:53.280766 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 3122
2026-09-25T14:56:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2714 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:56:53 dist-test-slave-v5vl krb5kdc[3650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:56:53 dist-test-slave-v5vl krb5kdc[3650](info): set up 2 sockets
Sep 25 14:56:53 dist-test-slave-v5vl krb5kdc[3650](info): commencing operation
krb5kdc: starting...
W20260925 14:56:54.368692 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:56:54 dist-test-slave-v5vl krb5kdc[3650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:56:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:56:54.595136 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37883
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:37883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:54.921600 3663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:54.921912 3663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:54.922112 3663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:54.929366 3663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:54.929471 3663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:54.929584 3663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:54.929687 3663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:54.929770 3663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:54.939498 3663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:37883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:54.940864 3663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:54.942099 3663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:54.953217 3668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:54.953238 3669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:54.953312 3671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:54.954037 3663 server_base.cc:1034] running on GCE node
I20260925 14:56:54.954787 3663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:54.956050 3663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:54.957237 3663 hybrid_clock.cc:648] HybridClock initialized: now 1790348214957190 us; error 36 us; skew 500 ppm
I20260925 14:56:54.957562 3663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:54 dist-test-slave-v5vl krb5kdc[3650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348214, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:54.960492 3663 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:56:54.961850 3663 webserver.cc:460] Webserver started at http://127.17.133.254:41431/ using document root <none> and password file <none>
I20260925 14:56:54.962502 3663 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:54.962657 3663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:54.962956 3663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:54.965764 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "9a3f828139e84080b9fad517abb17745"
format_stamp: "Formatted at 2026-09-25 14:56:54 on dist-test-slave-v5vl"
server_key: "689d9cf63e76800855f9d9fad85105ec"
server_key_iv: "a0ca0a81ad5ba56c500eb95215a679cb"
server_key_version: "encryptionkey@0"
I20260925 14:56:54.966460 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "9a3f828139e84080b9fad517abb17745"
format_stamp: "Formatted at 2026-09-25 14:56:54 on dist-test-slave-v5vl"
server_key: "689d9cf63e76800855f9d9fad85105ec"
server_key_iv: "a0ca0a81ad5ba56c500eb95215a679cb"
server_key_version: "encryptionkey@0"
I20260925 14:56:54.971345 3663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 14:56:54.974373 3678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:54.975492 3663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:56:54.975661 3663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "9a3f828139e84080b9fad517abb17745"
format_stamp: "Formatted at 2026-09-25 14:56:54 on dist-test-slave-v5vl"
server_key: "689d9cf63e76800855f9d9fad85105ec"
server_key_iv: "a0ca0a81ad5ba56c500eb95215a679cb"
server_key_version: "encryptionkey@0"
I20260925 14:56:54.975836 3663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:54.987656 3663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:54.991026 3663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:54.991276 3663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:55.010063 3663 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:37883
I20260925 14:56:55.010071 3729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:37883 every 8 connection(s)
I20260925 14:56:55.011482 3663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:55.015188 3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:55.018003 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 3663
I20260925 14:56:55.018196 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:56:55.018644 17943 external_mini_cluster.cc:1428] Setting key 42b7b6dc145caa227fd3f3d0f27b2fc6
I20260925 14:56:55.026108 3730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745: Bootstrap starting.
I20260925 14:56:55.029042 3730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:55.030102 3730 log.cc:824] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745: Log is configured to *not* fsync() on all Append() calls
Sep 25 14:56:55 dist-test-slave-v5vl krb5kdc[3650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:55.033154 3730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745: No bootstrap required, opened a new log
I20260925 14:56:55.041373 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:55.020439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.17.133.254:37883)
0925 14:56:55.021253 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:55.021267 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:55.021316 (+ 49us) server_negotiation.cc:407] Connection header received
0925 14:56:55.022966 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:55.022985 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:55.023342 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:55.023719 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:55.025248 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:55.026598 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:55.027884 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:55.028530 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:55.028745 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:55.033905 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:55.033913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:55.033925 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:55.033986 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:55.037152 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:55.037967 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:55.037973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:55.037979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:55.038164 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:55.038567 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:55.038570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:55.038572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:55.039052 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:55.039366 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:55.039718 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 14:56:55.040069 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":386,"threads_started":1}
I20260925 14:56:55.042397 3730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3f828139e84080b9fad517abb17745" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37883 } }
I20260925 14:56:55.042759 3730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:55.042873 3730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3f828139e84080b9fad517abb17745, State: Initialized, Role: FOLLOWER
I20260925 14:56:55.043401 3730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3f828139e84080b9fad517abb17745" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37883 } }
I20260925 14:56:55.043668 3730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:55.043813 3730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:55.043984 3730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:55.046139 3730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3f828139e84080b9fad517abb17745" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37883 } }
I20260925 14:56:55.046600 3730 leader_election.cc:302] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a3f828139e84080b9fad517abb17745; no voters:
I20260925 14:56:55.047161 3730 leader_election.cc:288] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:56:55.047288 3735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:55.047963 3735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3f828139e84080b9fad517abb17745, State: Running, Role: LEADER
I20260925 14:56:55.048516 3735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3f828139e84080b9fad517abb17745" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37883 } }
I20260925 14:56:55.049408 3730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:55.051242 3737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a3f828139e84080b9fad517abb17745. Latest consensus state: current_term: 1 leader_uuid: "9a3f828139e84080b9fad517abb17745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3f828139e84080b9fad517abb17745" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37883 } } }
I20260925 14:56:55.051750 3737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:55.051685 3736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a3f828139e84080b9fad517abb17745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3f828139e84080b9fad517abb17745" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37883 } } }
I20260925 14:56:55.052019 3736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745 [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:55.054343 3741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:55.059237 3741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:55.067808 3741 catalog_manager.cc:1347] Generated new cluster ID: 6183195a7e3547e196db8245bd5efb9b
I20260925 14:56:55.067926 3741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:55.080623 3741 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:55.081970 3741 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:55.089323 3741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745: Generated new TSK 0
I20260925 14:56:55.090013 3741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:56:55.157910 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37883
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:55.486654 3756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:55.486974 3756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:55.487154 3756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:55.494189 3756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:55.494346 3756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:55.494518 3756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:56:55.503531 3756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:55.504706 3756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:55.505901 3756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:55.518559 3762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:55.518509 3761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:55.518537 3764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:55.519397 3756 server_base.cc:1034] running on GCE node
I20260925 14:56:55.519968 3756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:55.520944 3756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:55.522161 3756 hybrid_clock.cc:648] HybridClock initialized: now 1790348215522104 us; error 58 us; skew 500 ppm
I20260925 14:56:55.522459 3756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:55 dist-test-slave-v5vl krb5kdc[3650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:55.525520 3756 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:56:55.526960 3756 webserver.cc:460] Webserver started at http://127.17.133.193:34609/ using document root <none> and password file <none>
I20260925 14:56:55.527626 3756 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:55.527766 3756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:55.528085 3756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:55.530874 3756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "0fcf55f07b3547988c0107fe667541ff"
format_stamp: "Formatted at 2026-09-25 14:56:55 on dist-test-slave-v5vl"
server_key: "b599c41f0175a3de4421ce4c1366e395"
server_key_iv: "d329fa473556457baed8286fe7ee730e"
server_key_version: "encryptionkey@0"
I20260925 14:56:55.531744 3756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "0fcf55f07b3547988c0107fe667541ff"
format_stamp: "Formatted at 2026-09-25 14:56:55 on dist-test-slave-v5vl"
server_key: "b599c41f0175a3de4421ce4c1366e395"
server_key_iv: "d329fa473556457baed8286fe7ee730e"
server_key_version: "encryptionkey@0"
I20260925 14:56:55.536981 3756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 14:56:55.540153 3771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:55.541689 3756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:55.541857 3756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "0fcf55f07b3547988c0107fe667541ff"
format_stamp: "Formatted at 2026-09-25 14:56:55 on dist-test-slave-v5vl"
server_key: "b599c41f0175a3de4421ce4c1366e395"
server_key_iv: "d329fa473556457baed8286fe7ee730e"
server_key_version: "encryptionkey@0"
I20260925 14:56:55.542047 3756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:55.570607 3756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:55.574198 3756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:55.574461 3756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:55.575304 3756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:55.576550 3756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:55.576666 3756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:55.576825 3756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:55.576905 3756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:55.607722 3756 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:35833
I20260925 14:56:55.607800 3883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:35833 every 8 connection(s)
I20260925 14:56:55.608954 3756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:55.614548 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 3756
I20260925 14:56:55.614750 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:56:55.615183 17943 external_mini_cluster.cc:1428] Setting key 9fb3ee352b5f89f46e0be466394cc9bf
Sep 25 14:56:55 dist-test-slave-v5vl krb5kdc[3650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348215, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:55.628425 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:55.611457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:40907 (local address 127.17.133.254:37883)
0925 14:56:55.611911 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:55.611916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:55.612704 (+ 788us) server_negotiation.cc:407] Connection header received
0925 14:56:55.614460 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:55.614466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:55.614563 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:55.614691 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:55.616276 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:55.617279 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:55.618767 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:55.619467 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:55.619618 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:55.622836 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:55.622841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:55.622844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:55.622873 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:55.625434 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:55.626065 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:55.626072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:55.626075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:55.626224 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:55.626637 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:55.626645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:55.626651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:55.627000 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:55.627150 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:55.627743 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 14:56:55.628002 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
I20260925 14:56:55.628969 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:55.611898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37883 (local address 127.17.133.193:40907)
0925 14:56:55.612508 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:55.612543 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:55.614201 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:55.614920 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:55.614928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:55.615388 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:55.616071 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:55.616113 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:55.617424 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:55.617428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:55.618626 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:55.618637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:55.619676 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:55.619680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:55.619868 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:55.620525 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:55.620546 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:55.622628 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:55.625567 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:55.625573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:55.625589 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:55.625956 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:55.626348 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:55.626351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:55.626353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:55.626528 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:55.627117 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:55.627122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:55.627396 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 14:56:55.627612 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 14:56:55.627980 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":255,"threads_started":1}
I20260925 14:56:55.630294 3884 heartbeater.cc:347] Connected to a master server at 127.17.133.254:37883
I20260925 14:56:55.630560 3884 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:55.631173 3884 heartbeater.cc:511] Master 127.17.133.254:37883 requested a full tablet report, sending...
I20260925 14:56:55.633095 3695 ts_manager.cc:194] Registered new tserver with Master: 0fcf55f07b3547988c0107fe667541ff (127.17.133.193:35833)
I20260925 14:56:55.634490 3695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:40907
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:56:55.667104 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37883
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:55.981256 3890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:55.981509 3890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:55.981604 3890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:55.988399 3890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:55.988498 3890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:55.988685 3890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:56:55.998526 3890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:55.999734 3890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:56.001024 3890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:56.014174 3895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:56.014372 3896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:56.014854 3898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:56.015532 3890 server_base.cc:1034] running on GCE node
I20260925 14:56:56.016161 3890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:56.017148 3890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:56.018379 3890 hybrid_clock.cc:648] HybridClock initialized: now 1790348216018301 us; error 76 us; skew 500 ppm
I20260925 14:56:56.018736 3890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:56 dist-test-slave-v5vl krb5kdc[3650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348216, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:56.022138 3890 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:56:56.025547 3890 webserver.cc:460] Webserver started at http://127.17.133.194:34973/ using document root <none> and password file <none>
I20260925 14:56:56.026161 3890 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:56.026285 3890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:56.026558 3890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:56.029341 3890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "a981e377b12d426f8231d2a0e005b725"
format_stamp: "Formatted at 2026-09-25 14:56:56 on dist-test-slave-v5vl"
server_key: "f91274ba5bb0a0f669c31770eb2f86a1"
server_key_iv: "ada28018a36f1353c847cb7a907c9419"
server_key_version: "encryptionkey@0"
I20260925 14:56:56.030038 3890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "a981e377b12d426f8231d2a0e005b725"
format_stamp: "Formatted at 2026-09-25 14:56:56 on dist-test-slave-v5vl"
server_key: "f91274ba5bb0a0f669c31770eb2f86a1"
server_key_iv: "ada28018a36f1353c847cb7a907c9419"
server_key_version: "encryptionkey@0"
I20260925 14:56:56.034662 3890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:56:56.037742 3905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:56.038926 3890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:56:56.039086 3890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "a981e377b12d426f8231d2a0e005b725"
format_stamp: "Formatted at 2026-09-25 14:56:56 on dist-test-slave-v5vl"
server_key: "f91274ba5bb0a0f669c31770eb2f86a1"
server_key_iv: "ada28018a36f1353c847cb7a907c9419"
server_key_version: "encryptionkey@0"
I20260925 14:56:56.039273 3890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:56.068233 3890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:56.072069 3890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:56.072360 3890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:56.073328 3890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:56.074630 3890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:56.074730 3890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:56.074872 3890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:56.074952 3890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:56.106385 3890 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:44971
I20260925 14:56:56.106467 4017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:44971 every 8 connection(s)
I20260925 14:56:56.107640 3890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:56.109282 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 3890
I20260925 14:56:56.109467 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:56:56.109956 17943 external_mini_cluster.cc:1428] Setting key d3385e90719a8adc43e93d5ac105ac8b
Sep 25 14:56:56 dist-test-slave-v5vl krb5kdc[3650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348216, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:56.127207 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.110173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:35641 (local address 127.17.133.254:37883)
0925 14:56:56.110391 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.110397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.111642 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 14:56:56.113541 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.113546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.113638 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.113740 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:56.115095 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.115782 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.117393 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.117813 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.117946 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.121799 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:56.121804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:56.121807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:56.121837 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:56.124267 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:56.125014 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:56.125024 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:56.125026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:56.125160 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:56.125617 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:56.125625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:56.125627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:56.125866 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:56.126015 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.126626 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.126866 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:56:56.128155 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.110723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37883 (local address 127.17.133.194:35641)
0925 14:56:56.111500 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:56.111544 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:56.113295 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:56.113962 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:56.113970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:56.114424 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:56.114933 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.114949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.115932 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.115937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.117253 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.117264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.118032 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.118039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.118271 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.118949 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:56.118981 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:56.121590 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:56.124411 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:56.124419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:56.124433 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:56.124891 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:56.125268 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:56.125274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:56.125277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:56.125483 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:56.125999 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:56.126006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:56.126300 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 14:56:56.126530 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 14:56:56.127022 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":337,"threads_started":1}
I20260925 14:56:56.129752 4018 heartbeater.cc:347] Connected to a master server at 127.17.133.254:37883
I20260925 14:56:56.130045 4018 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:56.130719 4018 heartbeater.cc:511] Master 127.17.133.254:37883 requested a full tablet report, sending...
I20260925 14:56:56.132210 3695 ts_manager.cc:194] Registered new tserver with Master: a981e377b12d426f8231d2a0e005b725 (127.17.133.194:44971)
I20260925 14:56:56.133424 3695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:35641
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:56:56.163107 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37883
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:56.486604 4023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:56.486887 4023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:56.487016 4023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:56.494613 4023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:56.494740 4023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:56.494923 4023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:56:56.504890 4023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:56.506148 4023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:56.507443 4023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:56.520604 4029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:56.520646 4031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:56.520536 4028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:56.521363 4023 server_base.cc:1034] running on GCE node
I20260925 14:56:56.521991 4023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:56.522974 4023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:56.524194 4023 hybrid_clock.cc:648] HybridClock initialized: now 1790348216524151 us; error 71 us; skew 500 ppm
I20260925 14:56:56.524492 4023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:56:56 dist-test-slave-v5vl krb5kdc[3650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348216, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:56.527482 4023 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:56:56.528967 4023 webserver.cc:460] Webserver started at http://127.17.133.195:36929/ using document root <none> and password file <none>
I20260925 14:56:56.529634 4023 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:56.529775 4023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:56.530092 4023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:56.533152 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "41f8fd5c138b41de8f62848af5831958"
format_stamp: "Formatted at 2026-09-25 14:56:56 on dist-test-slave-v5vl"
server_key: "0decbba82fdc702e9e407b20cca62da6"
server_key_iv: "986f19d701fb2d9da33b898e8bc270ff"
server_key_version: "encryptionkey@0"
I20260925 14:56:56.533931 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "41f8fd5c138b41de8f62848af5831958"
format_stamp: "Formatted at 2026-09-25 14:56:56 on dist-test-slave-v5vl"
server_key: "0decbba82fdc702e9e407b20cca62da6"
server_key_iv: "986f19d701fb2d9da33b898e8bc270ff"
server_key_version: "encryptionkey@0"
I20260925 14:56:56.539292 4023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:56:56.542819 4038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:56.544245 4023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:56:56.544430 4023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "41f8fd5c138b41de8f62848af5831958"
format_stamp: "Formatted at 2026-09-25 14:56:56 on dist-test-slave-v5vl"
server_key: "0decbba82fdc702e9e407b20cca62da6"
server_key_iv: "986f19d701fb2d9da33b898e8bc270ff"
server_key_version: "encryptionkey@0"
I20260925 14:56:56.544667 4023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:56.562000 4023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:56.565932 4023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:56.566237 4023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:56.567165 4023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:56.568446 4023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:56.568513 4023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:56.568842 4023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:56.568899 4023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:56.601092 4023 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:34033
I20260925 14:56:56.601140 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:34033 every 8 connection(s)
I20260925 14:56:56.602370 4023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:56.609341 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 4023
I20260925 14:56:56.609529 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:56:56.610136 17943 external_mini_cluster.cc:1428] Setting key 27c6918205f65a04b46a510ae68c078c
Sep 25 14:56:56 dist-test-slave-v5vl krb5kdc[3650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348216, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:56:56.622174 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.604912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:42529 (local address 127.17.133.254:37883)
0925 14:56:56.605126 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.605132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.606176 (+ 1044us) server_negotiation.cc:407] Connection header received
0925 14:56:56.607952 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.607958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.608090 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.608214 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:56.610237 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.611451 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.613426 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.613844 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.613957 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.617311 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:56.617316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:56.617319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:56.617349 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:56.619676 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:56.620289 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:56.620292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:56.620295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:56.620412 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:56.620819 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:56.620822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:56.620824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:56.621068 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:56.621189 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.621795 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.621945 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 14:56:56.623030 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.605354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37883 (local address 127.17.133.195:42529)
0925 14:56:56.606006 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:56.606041 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:56.607688 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:56.608390 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:56.608398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:56.609236 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:56.610042 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.610066 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.611646 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.611651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.613287 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.613298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.614039 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.614043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.614268 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.614960 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:56.614986 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:56.617138 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:56.619794 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:56.619800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:56.619811 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:56.620193 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:56.620500 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:56.620503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:56.620505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:56.620723 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:56.621167 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:56.621172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:56.621483 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 14:56:56.621700 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 14:56:56.622031 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":255,"threads_started":1}
I20260925 14:56:56.624464 4151 heartbeater.cc:347] Connected to a master server at 127.17.133.254:37883
I20260925 14:56:56.624815 4151 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:56.625620 4151 heartbeater.cc:511] Master 127.17.133.254:37883 requested a full tablet report, sending...
I20260925 14:56:56.626883 3694 ts_manager.cc:194] Registered new tserver with Master: 41f8fd5c138b41de8f62848af5831958 (127.17.133.195:34033)
I20260925 14:56:56.627672 3694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:42529
I20260925 14:56:56.637351 3884 heartbeater.cc:503] Master 127.17.133.254:37883 was elected leader, sending a full tablet report...
I20260925 14:56:56.637846 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:56.654660 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.641735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.17.133.254:37883)
0925 14:56:56.641958 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.641964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.642228 (+ 264us) server_negotiation.cc:407] Connection header received
0925 14:56:56.642408 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.642412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.642535 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.642657 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:56.644250 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.645083 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.646353 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.646829 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.646919 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.649596 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:56.649602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:56.649607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:56.649645 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:56.652086 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:56.652808 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:56.652812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:56.652814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:56.652934 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:56.653302 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:56.653305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:56.653307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:56.653589 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:56.653784 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.654128 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.654333 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:56:56.658612 3694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:56.665395 3694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:56.686931 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.676430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:35833 (local address 127.0.0.1:59780)
0925 14:56:56.677544 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:56.677617 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:56.677824 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:56.679379 (+ 1555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:56.679387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:56.679421 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:56.679939 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.679954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.682066 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.682073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.684509 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.684525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.685884 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.685890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.686071 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.686134 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 14:56:56.686268 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 14:56:56.686466 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":38,"thread_start_us":2067,"threads_started":1}
I20260925 14:56:56.687589 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.676729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.17.133.193:35833)
0925 14:56:56.678521 (+ 1792us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.678532 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.678564 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:56:56.678678 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.678687 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.678951 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.679138 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:56.680219 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.681919 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.684705 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.685669 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.685887 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.686080 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.686311 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.686484 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1604,"thread_start_us":1463,"threads_started":1}
I20260925 14:56:56.687623 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.676550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:44971 (local address 127.0.0.1:53758)
0925 14:56:56.678783 (+ 2233us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:56.678804 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:56.678907 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:56.680512 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:56.680516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:56.680543 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:56.680941 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.680953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.682689 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.682696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.684635 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.684651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.686762 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.686768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.686941 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.686971 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 14:56:56.687047 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:56:56.687173 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2111,"mutex_wait_us":40,"thread_start_us":1522,"threads_started":1}
I20260925 14:56:56.688340 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.677121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.17.133.194:44971)
0925 14:56:56.679938 (+ 2817us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.679946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.679972 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:56:56.680082 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.680091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.680298 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.681144 (+ 846us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:56.681344 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.682567 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.684797 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.686284 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.686443 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.686617 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.687927 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.688106 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2662,"thread_start_us":2527,"threads_started":1}
I20260925 14:56:56.688900 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.677149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:34033 (local address 127.0.0.1:56920)
0925 14:56:56.680296 (+ 3147us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:56.680317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:56.680438 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:56.681289 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:56.681292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:56.681309 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:56.681566 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.681573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.683452 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.683457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.685695 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.685709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.688134 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.688140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.688310 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.688340 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 14:56:56.688425 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 14:56:56.688541 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3040,"mutex_wait_us":48,"thread_start_us":2929,"threads_started":1}
I20260925 14:56:56.689138 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.677767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.17.133.195:34033)
0925 14:56:56.679508 (+ 1741us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.679518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.680555 (+ 1037us) server_negotiation.cc:407] Connection header received
0925 14:56:56.680691 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.680698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.680902 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.681072 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:56.681731 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.683264 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.685885 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.687611 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.687800 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.687961 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.688669 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.688832 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1533,"thread_start_us":1384,"threads_started":1}
I20260925 14:56:56.691838 3953 tablet_service.cc:1467] Processing CreateTablet for tablet f287b99fe8ac48ffaf354ee23b46cd5f (DEFAULT_TABLE table=test-table [id=f4202978a3b549ef8cc0e2d6b94f936c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:56.691834 3819 tablet_service.cc:1467] Processing CreateTablet for tablet f287b99fe8ac48ffaf354ee23b46cd5f (DEFAULT_TABLE table=test-table [id=f4202978a3b549ef8cc0e2d6b94f936c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:56.692468 4086 tablet_service.cc:1467] Processing CreateTablet for tablet f287b99fe8ac48ffaf354ee23b46cd5f (DEFAULT_TABLE table=test-table [id=f4202978a3b549ef8cc0e2d6b94f936c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:56.693171 3953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f287b99fe8ac48ffaf354ee23b46cd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:56.693243 3819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f287b99fe8ac48ffaf354ee23b46cd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:56.693758 4086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f287b99fe8ac48ffaf354ee23b46cd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:56.704042 4169 tablet_bootstrap.cc:492] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: Bootstrap starting.
I20260925 14:56:56.707250 4169 tablet_bootstrap.cc:654] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:56.707471 4170 tablet_bootstrap.cc:492] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: Bootstrap starting.
I20260925 14:56:56.708223 4171 tablet_bootstrap.cc:492] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: Bootstrap starting.
I20260925 14:56:56.708699 4169 log.cc:824] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:56.711413 4169 tablet_bootstrap.cc:492] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: No bootstrap required, opened a new log
I20260925 14:56:56.711673 4169 ts_tablet_manager.cc:1397] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 14:56:56.711786 4171 tablet_bootstrap.cc:654] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:56.712548 4170 tablet_bootstrap.cc:654] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:56.713023 4171 log.cc:824] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:56.713755 4170 log.cc:824] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:56.716027 4171 tablet_bootstrap.cc:492] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: No bootstrap required, opened a new log
I20260925 14:56:56.716270 4171 ts_tablet_manager.cc:1397] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 14:56:56.717125 4170 tablet_bootstrap.cc:492] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: No bootstrap required, opened a new log
I20260925 14:56:56.717373 4170 ts_tablet_manager.cc:1397] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260925 14:56:56.720157 4169 raft_consensus.cc:348] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } }
I20260925 14:56:56.720497 4169 raft_consensus.cc:374] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:56.720669 4169 raft_consensus.cc:729] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a981e377b12d426f8231d2a0e005b725, State: Initialized, Role: FOLLOWER
I20260925 14:56:56.721279 4169 consensus_queue.cc:260] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } }
I20260925 14:56:56.722373 4018 heartbeater.cc:503] Master 127.17.133.254:37883 was elected leader, sending a full tablet report...
I20260925 14:56:56.722854 4169 ts_tablet_manager.cc:1428] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 14:56:56.726809 4171 raft_consensus.cc:348] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } }
I20260925 14:56:56.727247 4171 raft_consensus.cc:374] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:56.727341 4171 raft_consensus.cc:729] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcf55f07b3547988c0107fe667541ff, State: Initialized, Role: FOLLOWER
I20260925 14:56:56.727959 4171 consensus_queue.cc:260] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } }
I20260925 14:56:56.729714 4171 ts_tablet_manager.cc:1428] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 14:56:56.730103 4170 raft_consensus.cc:348] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } }
I20260925 14:56:56.730546 4170 raft_consensus.cc:374] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:56.730674 4170 raft_consensus.cc:729] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f8fd5c138b41de8f62848af5831958, State: Initialized, Role: FOLLOWER
I20260925 14:56:56.731261 4170 consensus_queue.cc:260] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } }
I20260925 14:56:56.732297 4151 heartbeater.cc:503] Master 127.17.133.254:37883 was elected leader, sending a full tablet report...
I20260925 14:56:56.732971 4170 ts_tablet_manager.cc:1428] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260925 14:56:56.853945 4152 tablet.cc:2369] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:56:56.859616 4019 tablet.cc:2369] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:56:56.861433 3885 tablet.cc:2369] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:56.868026 4177 raft_consensus.cc:482] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:56.868338 4177 raft_consensus.cc:504] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } }
I20260925 14:56:56.870342 4177 leader_election.cc:288] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fcf55f07b3547988c0107fe667541ff (127.17.133.193:35833), a981e377b12d426f8231d2a0e005b725 (127.17.133.194:44971)
I20260925 14:56:56.875934 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.870768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:59451 (local address 127.17.133.193:35833)
0925 14:56:56.870982 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.870989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.871124 (+ 135us) server_negotiation.cc:407] Connection header received
0925 14:56:56.871356 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.871363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.871454 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.871557 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:56.871942 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.872752 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.874327 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.875085 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.875250 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.875331 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.875577 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.875697 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:56:56.875960 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.870787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:35833 (local address 127.17.133.195:59451)
0925 14:56:56.871015 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:56.871039 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:56.871185 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:56.871619 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:56.871622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:56.871646 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:56.871797 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.871805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.872889 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.872901 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.874173 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.874183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.875238 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.875243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.875427 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.875450 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:56:56.875552 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:56:56.875683 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":23}
I20260925 14:56:56.876749 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.870876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:46735 (local address 127.17.133.194:44971)
0925 14:56:56.871128 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.871134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.871836 (+ 702us) server_negotiation.cc:407] Connection header received
0925 14:56:56.872016 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.872025 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.872083 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.872178 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:56.872727 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.873404 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.875013 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.875783 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.875962 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.876030 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.876365 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.876500 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 14:56:56.876825 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.870784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:44971 (local address 127.17.133.195:46735)
0925 14:56:56.871731 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:56.871752 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:56.871887 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:56.872191 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:56.872196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:56.872218 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:56.872538 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.872546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.873546 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.873551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.874861 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:56.874872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.875982 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.875988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:56.876200 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.876228 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 14:56:56.876312 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:56:56.876427 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":148,"thread_start_us":420,"threads_started":1}
I20260925 14:56:56.877063 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f287b99fe8ac48ffaf354ee23b46cd5f" candidate_uuid: "41f8fd5c138b41de8f62848af5831958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fcf55f07b3547988c0107fe667541ff" is_pre_election: true
I20260925 14:56:56.877542 3839 raft_consensus.cc:2454] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41f8fd5c138b41de8f62848af5831958 in term 0.
I20260925 14:56:56.877715 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f287b99fe8ac48ffaf354ee23b46cd5f" candidate_uuid: "41f8fd5c138b41de8f62848af5831958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a981e377b12d426f8231d2a0e005b725" is_pre_election: true
I20260925 14:56:56.878201 3973 raft_consensus.cc:2454] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41f8fd5c138b41de8f62848af5831958 in term 0.
I20260925 14:56:56.878211 4042 leader_election.cc:302] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcf55f07b3547988c0107fe667541ff, 41f8fd5c138b41de8f62848af5831958; no voters:
I20260925 14:56:56.878630 4177 raft_consensus.cc:2781] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:56.878788 4177 raft_consensus.cc:482] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:56.878906 4177 raft_consensus.cc:3037] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:56.881103 4177 raft_consensus.cc:504] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } }
I20260925 14:56:56.881651 4177 leader_election.cc:288] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [CANDIDATE]: Term 1 election: Requested vote from peers 0fcf55f07b3547988c0107fe667541ff (127.17.133.193:35833), a981e377b12d426f8231d2a0e005b725 (127.17.133.194:44971)
I20260925 14:56:56.882050 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f287b99fe8ac48ffaf354ee23b46cd5f" candidate_uuid: "41f8fd5c138b41de8f62848af5831958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fcf55f07b3547988c0107fe667541ff"
I20260925 14:56:56.882074 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f287b99fe8ac48ffaf354ee23b46cd5f" candidate_uuid: "41f8fd5c138b41de8f62848af5831958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a981e377b12d426f8231d2a0e005b725"
I20260925 14:56:56.882236 3973 raft_consensus.cc:3037] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:56.882247 3839 raft_consensus.cc:3037] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:56.884300 3839 raft_consensus.cc:2454] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41f8fd5c138b41de8f62848af5831958 in term 1.
I20260925 14:56:56.884315 3973 raft_consensus.cc:2454] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41f8fd5c138b41de8f62848af5831958 in term 1.
I20260925 14:56:56.884685 4042 leader_election.cc:302] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcf55f07b3547988c0107fe667541ff, 41f8fd5c138b41de8f62848af5831958; no voters:
I20260925 14:56:56.884969 4177 raft_consensus.cc:2781] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:56.885391 4177 raft_consensus.cc:686] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 1 LEADER]: Becoming Leader. State: Replica: 41f8fd5c138b41de8f62848af5831958, State: Running, Role: LEADER
I20260925 14:56:56.885831 4177 consensus_queue.cc:237] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } }
I20260925 14:56:56.890272 3694 catalog_manager.cc:5495] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41f8fd5c138b41de8f62848af5831958 (127.17.133.195). New cstate: current_term: 1 leader_uuid: "41f8fd5c138b41de8f62848af5831958" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41f8fd5c138b41de8f62848af5831958" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 34033 } health_report { overall_health: HEALTHY } } }
I20260925 14:56:56.927491 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.922070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.17.133.195:34033)
0925 14:56:56.922272 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.922277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.922297 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:56:56.922431 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.922434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.922537 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.922695 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:56.923109 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.923821 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.925301 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.925698 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.925827 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.926402 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:56.926899 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:56.927080 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.927225 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.927270 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:56:56.935277 3973 raft_consensus.cc:1261] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [term 1 FOLLOWER]: Refusing update from remote peer 41f8fd5c138b41de8f62848af5831958: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:56.935354 3839 raft_consensus.cc:1261] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [term 1 FOLLOWER]: Refusing update from remote peer 41f8fd5c138b41de8f62848af5831958: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:56.936247 4179 consensus_queue.cc:1035] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fcf55f07b3547988c0107fe667541ff" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:56.936625 4177 consensus_queue.cc:1035] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a981e377b12d426f8231d2a0e005b725" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:56.947783 4182 mvcc.cc:205] Tried to move back new op lower bound from 7333266296557744128 to 7333266296367808512. Current Snapshot: MvccSnapshot[applied={T|T < 7333266296557744128}]
I20260925 14:56:56.959798 3694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:56.960007 3694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:56.963414 3694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a3f828139e84080b9fad517abb17745: Sending DeleteTablet for 3 replicas of tablet f287b99fe8ac48ffaf354ee23b46cd5f
I20260925 14:56:56.964542 3819 tablet_service.cc:1514] Processing DeleteTablet for tablet f287b99fe8ac48ffaf354ee23b46cd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:56 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:59780
I20260925 14:56:56.964542 3953 tablet_service.cc:1514] Processing DeleteTablet for tablet f287b99fe8ac48ffaf354ee23b46cd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:56 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:53758
I20260925 14:56:56.965214 4086 tablet_service.cc:1514] Processing DeleteTablet for tablet f287b99fe8ac48ffaf354ee23b46cd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:56 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:56920
I20260925 14:56:56.968887 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 3756
I20260925 14:56:56.973042 4193 tablet_replica.cc:330] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: stopping tablet replica
I20260925 14:56:56.973487 4193 raft_consensus.cc:2229] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:56.974241 4193 raft_consensus.cc:2258] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:56.977033 4192 tablet_replica.cc:330] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: stopping tablet replica
I20260925 14:56:56.977649 4192 raft_consensus.cc:2229] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:56.978281 4192 raft_consensus.cc:2258] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:56.978870 4193 ts_tablet_manager.cc:1905] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:56.980993 4191 tablet_replica.cc:330] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: stopping tablet replica
I20260925 14:56:56.981488 4192 ts_tablet_manager.cc:1905] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:56.985785 4191 raft_consensus.cc:2229] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:56.986295 4191 raft_consensus.cc:2258] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:56.986814 4193 ts_tablet_manager.cc:1918] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:56.986981 4193 log.cc:1196] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/f287b99fe8ac48ffaf354ee23b46cd5f
I20260925 14:56:56.987535 4193 ts_tablet_manager.cc:1939] T f287b99fe8ac48ffaf354ee23b46cd5f P 41f8fd5c138b41de8f62848af5831958: Deleting consensus metadata
I20260925 14:56:56.988754 4192 ts_tablet_manager.cc:1918] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:56.988984 4192 log.cc:1196] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/f287b99fe8ac48ffaf354ee23b46cd5f
I20260925 14:56:56.989347 3679 catalog_manager.cc:4841] TS 41f8fd5c138b41de8f62848af5831958 (127.17.133.195:34033): tablet f287b99fe8ac48ffaf354ee23b46cd5f (table test-table [id=f4202978a3b549ef8cc0e2d6b94f936c]) successfully deleted
I20260925 14:56:56.989641 4192 ts_tablet_manager.cc:1939] T f287b99fe8ac48ffaf354ee23b46cd5f P 0fcf55f07b3547988c0107fe667541ff: Deleting consensus metadata
Sep 25 14:56:56 dist-test-slave-v5vl krb5kdc[3650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:56:56.991402 3682 catalog_manager.cc:4841] TS 0fcf55f07b3547988c0107fe667541ff (127.17.133.193:35833): tablet f287b99fe8ac48ffaf354ee23b46cd5f (table test-table [id=f4202978a3b549ef8cc0e2d6b94f936c]) successfully deleted
I20260925 14:56:56.991600 4191 ts_tablet_manager.cc:1905] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:56.997105 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:56.969919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.17.133.193:35833)
0925 14:56:56.970099 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:56.970107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:56.974998 (+ 4891us) server_negotiation.cc:407] Connection header received
0925 14:56:56.976884 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:56.976891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:56.976971 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:56.977057 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:56.980884 (+ 3827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.981942 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.983856 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:56.984507 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:56.984622 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:56.991129 (+ 6507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:56.991140 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:56.991157 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:56.991207 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:56.994231 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:56.995016 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:56.995022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:56.995024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:56.995157 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:56.995536 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:56.995562 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:56.995565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:56.995883 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:56.996221 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:56.996467 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 14:56:56.996711 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:56:56.998087 3879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:56.998420 4191 ts_tablet_manager.cc:1918] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:56.998648 4191 log.cc:1196] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/f287b99fe8ac48ffaf354ee23b46cd5f
I20260925 14:56:56.999253 4191 ts_tablet_manager.cc:1939] T f287b99fe8ac48ffaf354ee23b46cd5f P a981e377b12d426f8231d2a0e005b725: Deleting consensus metadata
I20260925 14:56:57.001226 3680 catalog_manager.cc:4841] TS a981e377b12d426f8231d2a0e005b725 (127.17.133.194:44971): tablet f287b99fe8ac48ffaf354ee23b46cd5f (table test-table [id=f4202978a3b549ef8cc0e2d6b94f936c]) successfully deleted
I20260925 14:56:57.101317 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 3756
I20260925 14:56:57.125561 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 3890
Sep 25 14:56:57 dist-test-slave-v5vl krb5kdc[3650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:56:57.142431 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:57.126395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.17.133.194:44971)
0925 14:56:57.126561 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:57.126567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:57.126585 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:56:57.126712 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:57.126716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:57.126782 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:57.126865 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:57.128459 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:57.129267 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:57.131096 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:57.131464 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:57.131558 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:57.136792 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:57.136805 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:57.136820 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:57.136859 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:57.139603 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:57.140252 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:57.140260 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:57.140262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:57.140374 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:57.140811 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:57.140817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:57.140821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:57.141235 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:57.141540 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:57.141865 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 14:56:57.142096 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:56:57.143206 4013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:57.233139 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 3890
I20260925 14:56:57.250361 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 4023
Sep 25 14:56:57 dist-test-slave-v5vl krb5kdc[3650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:56:57.266461 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:57.251331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.17.133.195:34033)
0925 14:56:57.251537 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:57.251543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:57.251560 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:56:57.251624 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:57.251628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:57.251691 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:57.251805 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:57.253099 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:57.253833 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:57.255102 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:57.255529 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:57.255679 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:57.260573 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:57.260614 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:57.260637 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:57.260668 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:57.263629 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:57.264399 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:57.264405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:57.264408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:57.264550 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:57.265052 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:57.265055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:57.265057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:57.265372 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:57.265611 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:57.265935 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 14:56:57.266091 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:56:57.267127 4146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:57.390326 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 4023
I20260925 14:56:57.407285 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 3663
I20260925 14:56:57.408284 3725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:57.517342 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 3663
2026-09-25T14:56:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4237 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T14:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:57Z Disabled control of system clock
I20260925 14:56:57.590816 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37055
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:37055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:56:57.915859 4202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:57.916162 4202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:57.916244 4202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:57.923717 4202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:57.923866 4202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:57.923960 4202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:57.924065 4202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:57.924160 4202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:57.933404 4202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:37055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:57.934618 4202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:57.936276 4202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:57.948410 4208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:57.948436 4207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:57.948906 4210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:57.949488 4202 server_base.cc:1034] running on GCE node
I20260925 14:56:57.950259 4202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:57.951694 4202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:57.952886 4202 hybrid_clock.cc:648] HybridClock initialized: now 1790348217952846 us; error 27 us; skew 500 ppm
I20260925 14:56:57.953235 4202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:56:57.955410 4202 webserver.cc:460] Webserver started at http://127.17.133.254:34819/ using document root <none> and password file <none>
I20260925 14:56:57.956094 4202 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:57.956207 4202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:57.956527 4202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:57.959441 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "560198a9b8cd4bbfb973294b8741233c"
format_stamp: "Formatted at 2026-09-25 14:56:57 on dist-test-slave-v5vl"
server_key: "9eaf215ed1257e32fdf041b3d4f9ee08"
server_key_iv: "c796773bde10670f6f15dc05b09245b8"
server_key_version: "encryptionkey@0"
I20260925 14:56:57.960301 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "560198a9b8cd4bbfb973294b8741233c"
format_stamp: "Formatted at 2026-09-25 14:56:57 on dist-test-slave-v5vl"
server_key: "9eaf215ed1257e32fdf041b3d4f9ee08"
server_key_iv: "c796773bde10670f6f15dc05b09245b8"
server_key_version: "encryptionkey@0"
I20260925 14:56:57.965648 4202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 14:56:57.968972 4216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:57.970276 4202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 14:56:57.970471 4202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "560198a9b8cd4bbfb973294b8741233c"
format_stamp: "Formatted at 2026-09-25 14:56:57 on dist-test-slave-v5vl"
server_key: "9eaf215ed1257e32fdf041b3d4f9ee08"
server_key_iv: "c796773bde10670f6f15dc05b09245b8"
server_key_version: "encryptionkey@0"
I20260925 14:56:57.970685 4202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:57.999239 4202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:58.000113 4202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:58.000406 4202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:58.018661 4202 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:37055
I20260925 14:56:58.018682 4267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:37055 every 8 connection(s)
I20260925 14:56:58.019942 4202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:58.022794 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 4202
I20260925 14:56:58.022989 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:56:58.023437 17943 external_mini_cluster.cc:1428] Setting key b4850b74fb0f5418d7da6b99fed3c422
I20260925 14:56:58.023650 4268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:58.037628 4268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c: Bootstrap starting.
I20260925 14:56:58.040766 4268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:58.042043 4268 log.cc:824] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:58.043157 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:58.025358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.17.133.254:37055)
0925 14:56:58.026352 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:58.026365 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:58.026418 (+ 53us) server_negotiation.cc:407] Connection header received
0925 14:56:58.028541 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:58.028575 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:58.029163 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:58.029585 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:58.030118 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:58.032071 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:58.033497 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:58.034578 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:58.034756 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:58.040380 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:58.040390 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:58.040409 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:58.041164 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:58.041292 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:58.041322 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:58.041425 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:58.041656 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 14:56:58.041987 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":318,"threads_started":1}
I20260925 14:56:58.044860 4268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c: No bootstrap required, opened a new log
I20260925 14:56:58.052312 4268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560198a9b8cd4bbfb973294b8741233c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37055 } }
I20260925 14:56:58.052726 4268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:58.052827 4268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560198a9b8cd4bbfb973294b8741233c, State: Initialized, Role: FOLLOWER
I20260925 14:56:58.053432 4268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560198a9b8cd4bbfb973294b8741233c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37055 } }
I20260925 14:56:58.053720 4268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:58.053889 4268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:58.054097 4268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:58.056077 4268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560198a9b8cd4bbfb973294b8741233c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37055 } }
I20260925 14:56:58.056561 4268 leader_election.cc:302] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 560198a9b8cd4bbfb973294b8741233c; no voters:
I20260925 14:56:58.057153 4268 leader_election.cc:288] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:56:58.057300 4273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:58.058076 4273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [term 1 LEADER]: Becoming Leader. State: Replica: 560198a9b8cd4bbfb973294b8741233c, State: Running, Role: LEADER
I20260925 14:56:58.058609 4273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560198a9b8cd4bbfb973294b8741233c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37055 } }
I20260925 14:56:58.059209 4268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:58.061339 4275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 560198a9b8cd4bbfb973294b8741233c. Latest consensus state: current_term: 1 leader_uuid: "560198a9b8cd4bbfb973294b8741233c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560198a9b8cd4bbfb973294b8741233c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37055 } } }
I20260925 14:56:58.061731 4275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:58.064276 4282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:58.064154 4274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "560198a9b8cd4bbfb973294b8741233c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560198a9b8cd4bbfb973294b8741233c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 37055 } } }
I20260925 14:56:58.064467 4274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:58.069221 4282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:58.078487 4282 catalog_manager.cc:1347] Generated new cluster ID: 28922256b61c4addbbfe8e0ac1c28279
I20260925 14:56:58.078610 4282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:58.096022 4282 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:58.097362 4282 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:58.113801 4282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c: Generated new TSK 0
I20260925 14:56:58.114612 4282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:56:58.125068 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37055
--builtin_ntp_servers=127.17.133.212:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:56:58.487290 4292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:58.487556 4292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:58.487649 4292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:58.494483 4292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:58.494563 4292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:58.494664 4292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:56:58.503345 4292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:58.504637 4292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:58.506219 4292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:58.518515 4300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:58.518589 4298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:58.518712 4297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:58.519359 4292 server_base.cc:1034] running on GCE node
I20260925 14:56:58.519939 4292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:58.520843 4292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:58.522049 4292 hybrid_clock.cc:648] HybridClock initialized: now 1790348218522002 us; error 26 us; skew 500 ppm
I20260925 14:56:58.522347 4292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:56:58.524413 4292 webserver.cc:460] Webserver started at http://127.17.133.193:42875/ using document root <none> and password file <none>
I20260925 14:56:58.525094 4292 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:58.525214 4292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:58.525508 4292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:58.528131 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "e2714c36fffb47f0a8be53ca291e41fd"
format_stamp: "Formatted at 2026-09-25 14:56:58 on dist-test-slave-v5vl"
server_key: "02c9f468423acc0f3c6a87c55b42991b"
server_key_iv: "62f2801a2b6dcf7df27bf2174927609a"
server_key_version: "encryptionkey@0"
I20260925 14:56:58.528959 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "e2714c36fffb47f0a8be53ca291e41fd"
format_stamp: "Formatted at 2026-09-25 14:56:58 on dist-test-slave-v5vl"
server_key: "02c9f468423acc0f3c6a87c55b42991b"
server_key_iv: "62f2801a2b6dcf7df27bf2174927609a"
server_key_version: "encryptionkey@0"
I20260925 14:56:58.533812 4292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 14:56:58.537063 4306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:58.538262 4292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 14:56:58.538443 4292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "e2714c36fffb47f0a8be53ca291e41fd"
format_stamp: "Formatted at 2026-09-25 14:56:58 on dist-test-slave-v5vl"
server_key: "02c9f468423acc0f3c6a87c55b42991b"
server_key_iv: "62f2801a2b6dcf7df27bf2174927609a"
server_key_version: "encryptionkey@0"
I20260925 14:56:58.538654 4292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:58.556743 4292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:58.557542 4292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:58.557838 4292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:58.558796 4292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:58.560173 4292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:58.560287 4292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:58.560420 4292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:58.560508 4292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:58.592978 4292 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:36177
I20260925 14:56:58.593077 4418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:36177 every 8 connection(s)
I20260925 14:56:58.594344 4292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:58.603474 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 4292
I20260925 14:56:58.603685 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:56:58.604225 17943 external_mini_cluster.cc:1428] Setting key 28e3de426810e6251640adef7168b331
I20260925 14:56:58.606602 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:58.596927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:36175 (local address 127.17.133.254:37055)
0925 14:56:58.597354 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:58.597360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:58.598585 (+ 1225us) server_negotiation.cc:407] Connection header received
0925 14:56:58.600227 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:58.600231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:58.600295 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:58.600389 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:58.601390 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:58.602095 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:58.603563 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:58.604191 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:58.604329 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:58.605728 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:58.605734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:58.605739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:58.605766 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:58.605834 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:58.605852 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:58.605949 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:58.606159 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 14:56:58.606253 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":192,"threads_started":1}
I20260925 14:56:58.606983 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37055
--builtin_ntp_servers=127.17.133.212:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:56:58.607582 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:58.597567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37055 (local address 127.17.133.193:36175)
0925 14:56:58.598416 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:58.598456 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:58.600003 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:58.600577 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:58.600636 (+ 59us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:58.600724 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:58.601254 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:58.601269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:58.602219 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:58.602223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:58.603425 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:58.603437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:58.604386 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:58.604390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:58.604687 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:58.605414 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:56:58.605416 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:58.605455 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:56:58.605533 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:58.605956 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:58.605965 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:58.605971 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:56:58.606148 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 14:56:58.606536 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":253,"threads_started":1}
I20260925 14:56:58.609113 4419 heartbeater.cc:347] Connected to a master server at 127.17.133.254:37055
I20260925 14:56:58.609404 4419 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:58.610145 4419 heartbeater.cc:511] Master 127.17.133.254:37055 requested a full tablet report, sending...
I20260925 14:56:58.612202 4233 ts_manager.cc:194] Registered new tserver with Master: e2714c36fffb47f0a8be53ca291e41fd (127.17.133.193:36177)
I20260925 14:56:58.613821 4233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.193:36175
W20260925 14:56:58.925045 4423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:58.925312 4423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:58.925396 4423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:58.932447 4423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:58.932534 4423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:58.932731 4423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:56:58.942453 4423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:58.943704 4423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:58.945492 4423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:58.958535 4429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:58.958673 4428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:58.959151 4431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:58.959695 4423 server_base.cc:1034] running on GCE node
I20260925 14:56:58.960275 4423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:58.961264 4423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:58.962472 4423 hybrid_clock.cc:648] HybridClock initialized: now 1790348218962425 us; error 40 us; skew 500 ppm
I20260925 14:56:58.962786 4423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:56:58.965090 4423 webserver.cc:460] Webserver started at http://127.17.133.194:41699/ using document root <none> and password file <none>
I20260925 14:56:58.965796 4423 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:58.965952 4423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:58.966279 4423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:58.969120 4423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "5213d009845340e68e57bce0d839dcb5"
format_stamp: "Formatted at 2026-09-25 14:56:58 on dist-test-slave-v5vl"
server_key: "313d1aa2f7e3c995aebdb8c4a745a47f"
server_key_iv: "63ce3068712e5398b2d25ee2fa18d503"
server_key_version: "encryptionkey@0"
I20260925 14:56:58.969952 4423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "5213d009845340e68e57bce0d839dcb5"
format_stamp: "Formatted at 2026-09-25 14:56:58 on dist-test-slave-v5vl"
server_key: "313d1aa2f7e3c995aebdb8c4a745a47f"
server_key_iv: "63ce3068712e5398b2d25ee2fa18d503"
server_key_version: "encryptionkey@0"
I20260925 14:56:58.974761 4423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:56:58.978214 4437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:58.979415 4423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:56:58.979598 4423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "5213d009845340e68e57bce0d839dcb5"
format_stamp: "Formatted at 2026-09-25 14:56:58 on dist-test-slave-v5vl"
server_key: "313d1aa2f7e3c995aebdb8c4a745a47f"
server_key_iv: "63ce3068712e5398b2d25ee2fa18d503"
server_key_version: "encryptionkey@0"
I20260925 14:56:58.979809 4423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:58.995453 4423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:58.996181 4423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:58.996459 4423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:58.997399 4423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:58.998761 4423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:58.998881 4423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:58.999011 4423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:58.999099 4423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:59.032050 4423 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:34931
I20260925 14:56:59.032142 4549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:34931 every 8 connection(s)
I20260925 14:56:59.033366 4423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:59.041050 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 4423
I20260925 14:56:59.041224 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:56:59.041688 17943 external_mini_cluster.cc:1428] Setting key 1b173088ddc9e3bf849792ee8d6f8e55
I20260925 14:56:59.044284 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37055
--builtin_ntp_servers=127.17.133.212:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:56:59.045982 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.035941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:40459 (local address 127.17.133.254:37055)
0925 14:56:59.036146 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.036152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.037639 (+ 1487us) server_negotiation.cc:407] Connection header received
0925 14:56:59.039300 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.039304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.039368 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.039454 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:59.040405 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.041234 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.043241 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.043649 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.043779 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.045172 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:59.045184 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:59.045187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:59.045205 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:59.045234 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:59.045245 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:59.045336 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.045589 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.045685 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:56:59.046994 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.036527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37055 (local address 127.17.133.194:40459)
0925 14:56:59.037510 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:59.037550 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:59.039070 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:59.039652 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:59.039660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:59.039742 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:59.040269 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.040284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.041393 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.041398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.043028 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.043039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.043873 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.043881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.044116 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.044848 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:56:59.044850 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:59.044888 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:56:59.044972 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:59.045346 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:59.045358 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:59.045366 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:56:59.045572 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 14:56:59.045978 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":276,"threads_started":1}
I20260925 14:56:59.048368 4550 heartbeater.cc:347] Connected to a master server at 127.17.133.254:37055
I20260925 14:56:59.048717 4550 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:59.049589 4550 heartbeater.cc:511] Master 127.17.133.254:37055 requested a full tablet report, sending...
I20260925 14:56:59.051103 4233 ts_manager.cc:194] Registered new tserver with Master: 5213d009845340e68e57bce0d839dcb5 (127.17.133.194:34931)
I20260925 14:56:59.051995 4233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.194:40459
W20260925 14:56:59.367777 4553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:59.368033 4553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:59.368117 4553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:59.375296 4553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:59.375398 4553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:59.375502 4553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:56:59.384549 4553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:56:59.385815 4553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:59.387585 4553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:59.400040 4561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:59.400276 4559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:59.400091 4558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:59.401614 4553 server_base.cc:1034] running on GCE node
I20260925 14:56:59.402143 4553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:59.403640 4553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:59.404886 4553 hybrid_clock.cc:648] HybridClock initialized: now 1790348219404818 us; error 37 us; skew 500 ppm
I20260925 14:56:59.405212 4553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:56:59.407267 4553 webserver.cc:460] Webserver started at http://127.17.133.195:37429/ using document root <none> and password file <none>
I20260925 14:56:59.407943 4553 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:59.408069 4553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:59.408392 4553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:59.411125 4553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "c618fc59c4ca4bb2b01593515479721f"
format_stamp: "Formatted at 2026-09-25 14:56:59 on dist-test-slave-v5vl"
server_key: "78b3b2eb5a7c445b312250a6373d992d"
server_key_iv: "6e738aeb4124ca4d9f772d5f8ede9415"
server_key_version: "encryptionkey@0"
I20260925 14:56:59.411886 4553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "c618fc59c4ca4bb2b01593515479721f"
format_stamp: "Formatted at 2026-09-25 14:56:59 on dist-test-slave-v5vl"
server_key: "78b3b2eb5a7c445b312250a6373d992d"
server_key_iv: "6e738aeb4124ca4d9f772d5f8ede9415"
server_key_version: "encryptionkey@0"
I20260925 14:56:59.416517 4553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 14:56:59.419741 4567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:59.420955 4553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:56:59.421120 4553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "c618fc59c4ca4bb2b01593515479721f"
format_stamp: "Formatted at 2026-09-25 14:56:59 on dist-test-slave-v5vl"
server_key: "78b3b2eb5a7c445b312250a6373d992d"
server_key_iv: "6e738aeb4124ca4d9f772d5f8ede9415"
server_key_version: "encryptionkey@0"
I20260925 14:56:59.421321 4553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:59.452981 4553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:59.453853 4553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:59.454159 4553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:59.455156 4553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:59.456748 4553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:59.456856 4553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:59.456960 4553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:59.457017 4553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:59.489444 4553 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:37677
I20260925 14:56:59.489549 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:37677 every 8 connection(s)
I20260925 14:56:59.490710 4553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:59.499593 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 4553
I20260925 14:56:59.499766 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:56:59.500298 17943 external_mini_cluster.cc:1428] Setting key 529998c170566e711b087a8c1d17b307
I20260925 14:56:59.502736 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.492919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:39569 (local address 127.17.133.254:37055)
0925 14:56:59.493133 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.493139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.494433 (+ 1294us) server_negotiation.cc:407] Connection header received
0925 14:56:59.496064 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.496069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.496134 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.496221 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:59.497188 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.497925 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.499556 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.500068 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.500161 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.501817 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:59.501822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:59.501825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:59.501842 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:59.501897 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:59.501913 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:59.502012 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.502290 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.502399 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:56:59.503707 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.493487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:37055 (local address 127.17.133.195:39569)
0925 14:56:59.494294 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:59.494331 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:59.495838 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:59.496407 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:59.496415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:59.496493 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:59.497052 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.497068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.498048 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.498051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.499413 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.499428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.500279 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.500288 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.500624 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.501477 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:56:59.501480 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:59.501520 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:56:59.501634 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:59.502031 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:59.502044 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:59.502052 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:56:59.502252 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 14:56:59.502558 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":242,"threads_started":1}
I20260925 14:56:59.505007 4680 heartbeater.cc:347] Connected to a master server at 127.17.133.254:37055
I20260925 14:56:59.505290 4680 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:59.506021 4680 heartbeater.cc:511] Master 127.17.133.254:37055 requested a full tablet report, sending...
I20260925 14:56:59.507300 4233 ts_manager.cc:194] Registered new tserver with Master: c618fc59c4ca4bb2b01593515479721f (127.17.133.195:37677)
I20260925 14:56:59.508085 4233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.195:39569
I20260925 14:56:59.515218 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:59.524814 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.519155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.17.133.254:37055)
0925 14:56:59.519316 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.519320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.519710 (+ 390us) server_negotiation.cc:407] Connection header received
0925 14:56:59.519835 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.519838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.519927 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.520010 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:59.520452 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.521211 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.522508 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.522907 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.522994 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.524060 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:59.524072 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:59.524075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:59.524092 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:59.524123 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:59.524135 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:59.524286 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.524486 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.524541 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:56:59.528795 4233 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:59.535516 4233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:59.577702 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.549354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.17.133.193:36177)
0925 14:56:59.566377 (+ 17023us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.566390 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.567208 (+ 818us) server_negotiation.cc:407] Connection header received
0925 14:56:59.567351 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.567364 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.568497 (+ 1133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.568755 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:59.569969 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.571396 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.574422 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.575705 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.575877 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.576118 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.577163 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.577346 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16787,"thread_start_us":16604,"threads_started":1}
I20260925 14:56:59.577773 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.548140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:34931 (local address 127.0.0.1:59936)
0925 14:56:59.563945 (+ 15805us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:59.563971 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:59.564144 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:59.570204 (+ 6060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:59.570209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:59.570232 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:59.570531 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.570539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.572432 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.572440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.574849 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.574863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.576974 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.576979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.577148 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.577182 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:56:59.577268 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:56:59.577388 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15581,"mutex_wait_us":2,"thread_start_us":16140,"threads_started":1}
I20260925 14:56:59.578154 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.548113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:37677 (local address 127.0.0.1:41108)
0925 14:56:59.555196 (+ 7083us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:59.555245 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:59.555451 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:59.570963 (+ 15512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:59.570969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:59.570997 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:59.571335 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.571346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.573186 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.573192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.575460 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.575473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.577292 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.577298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.577480 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.577508 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 14:56:59.577604 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 14:56:59.577750 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6766,"mutex_wait_us":22,"thread_start_us":6596,"threads_started":1}
I20260925 14:56:59.578169 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.548903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.17.133.195:37677)
0925 14:56:59.570436 (+ 21533us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.570444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.570473 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:56:59.570563 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.570571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.570761 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.570937 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:59.571536 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.573043 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.575667 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.577078 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.577306 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.577513 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.577644 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.577804 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21347,"thread_start_us":21185,"threads_started":1}
I20260925 14:56:59.578572 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.548753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.17.133.194:34931)
0925 14:56:59.569506 (+ 20753us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.569518 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.569552 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:56:59.569672 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.569684 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.569954 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.570686 (+ 732us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:59.570869 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.572299 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.575076 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.576446 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.576642 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.576851 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.578013 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.578173 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20555,"thread_start_us":20413,"threads_started":1}
I20260925 14:56:59.578905 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.549032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36177 (local address 127.0.0.1:42304)
0925 14:56:59.566874 (+ 17842us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:59.566892 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:59.567048 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:59.569457 (+ 2409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:59.569463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:59.569495 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:59.569789 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.569798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.571583 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.571589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.574220 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.574232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.575892 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.575899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.576463 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.576533 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 14:56:59.576683 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 14:56:59.576866 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17697,"mutex_wait_us":4,"thread_start_us":17515,"threads_started":1}
I20260925 14:56:59.582612 4485 tablet_service.cc:1467] Processing CreateTablet for tablet a619ecb6f74d410094ee0f32d1513fac (DEFAULT_TABLE table=test-table [id=20d67f96bf25451c95baf8315d13e871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:59.582614 4615 tablet_service.cc:1467] Processing CreateTablet for tablet a619ecb6f74d410094ee0f32d1513fac (DEFAULT_TABLE table=test-table [id=20d67f96bf25451c95baf8315d13e871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:59.582983 4354 tablet_service.cc:1467] Processing CreateTablet for tablet a619ecb6f74d410094ee0f32d1513fac (DEFAULT_TABLE table=test-table [id=20d67f96bf25451c95baf8315d13e871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:59.584149 4615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a619ecb6f74d410094ee0f32d1513fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:59.584152 4485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a619ecb6f74d410094ee0f32d1513fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:59.584316 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a619ecb6f74d410094ee0f32d1513fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:59.602919 4698 tablet_bootstrap.cc:492] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: Bootstrap starting.
I20260925 14:56:59.605480 4700 tablet_bootstrap.cc:492] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: Bootstrap starting.
I20260925 14:56:59.606256 4699 tablet_bootstrap.cc:492] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: Bootstrap starting.
I20260925 14:56:59.607923 4698 tablet_bootstrap.cc:654] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:59.609233 4698 log.cc:824] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:59.609470 4699 tablet_bootstrap.cc:654] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:59.609724 4700 tablet_bootstrap.cc:654] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:59.610682 4699 log.cc:824] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:59.610745 4700 log.cc:824] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:59.612762 4698 tablet_bootstrap.cc:492] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: No bootstrap required, opened a new log
I20260925 14:56:59.613091 4698 ts_tablet_manager.cc:1397] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 14:56:59.613526 4699 tablet_bootstrap.cc:492] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: No bootstrap required, opened a new log
I20260925 14:56:59.613754 4699 ts_tablet_manager.cc:1397] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 14:56:59.617074 4419 heartbeater.cc:503] Master 127.17.133.254:37055 was elected leader, sending a full tablet report...
I20260925 14:56:59.618573 4700 tablet_bootstrap.cc:492] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: No bootstrap required, opened a new log
I20260925 14:56:59.618820 4700 ts_tablet_manager.cc:1397] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260925 14:56:59.622069 4699 raft_consensus.cc:348] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } }
I20260925 14:56:59.622443 4699 raft_consensus.cc:374] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:59.622541 4699 raft_consensus.cc:729] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c618fc59c4ca4bb2b01593515479721f, State: Initialized, Role: FOLLOWER
I20260925 14:56:59.623245 4699 consensus_queue.cc:260] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } }
I20260925 14:56:59.625430 4680 heartbeater.cc:503] Master 127.17.133.254:37055 was elected leader, sending a full tablet report...
I20260925 14:56:59.626065 4699 ts_tablet_manager.cc:1428] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260925 14:56:59.625967 4698 raft_consensus.cc:348] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } }
I20260925 14:56:59.626343 4698 raft_consensus.cc:374] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:59.626430 4698 raft_consensus.cc:729] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2714c36fffb47f0a8be53ca291e41fd, State: Initialized, Role: FOLLOWER
I20260925 14:56:59.627043 4698 consensus_queue.cc:260] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } }
I20260925 14:56:59.629658 4700 raft_consensus.cc:348] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } }
I20260925 14:56:59.630182 4700 raft_consensus.cc:374] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:59.630300 4700 raft_consensus.cc:729] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5213d009845340e68e57bce0d839dcb5, State: Initialized, Role: FOLLOWER
I20260925 14:56:59.630970 4700 consensus_queue.cc:260] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } }
I20260925 14:56:59.632045 4550 heartbeater.cc:503] Master 127.17.133.254:37055 was elected leader, sending a full tablet report...
I20260925 14:56:59.632882 4700 ts_tablet_manager.cc:1428] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260925 14:56:59.634254 4698 ts_tablet_manager.cc:1428] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260925 14:56:59.679622 4704 raft_consensus.cc:482] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:59.679980 4704 raft_consensus.cc:504] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } }
I20260925 14:56:59.681754 4704 leader_election.cc:288] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5213d009845340e68e57bce0d839dcb5 (127.17.133.194:34931), e2714c36fffb47f0a8be53ca291e41fd (127.17.133.193:36177)
I20260925 14:56:59.687577 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.682403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:42345 (local address 127.17.133.193:36177)
0925 14:56:59.682640 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.682644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.682659 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:56:59.682722 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.682725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.682789 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.682898 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:59.683327 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.684134 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.685918 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.686712 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.686869 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.686941 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.687183 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.687305 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 14:56:59.687584 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.682230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:36177 (local address 127.17.133.195:42345)
0925 14:56:59.682437 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:59.682458 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:59.682596 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:59.682919 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:59.682923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:59.682955 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:59.683172 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.683180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.684292 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.684304 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.685767 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.685781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.686858 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.686863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.687030 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.687056 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 14:56:59.687157 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:56:59.687291 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":20}
I20260925 14:56:59.688407 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.682315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:38839 (local address 127.17.133.194:34931)
0925 14:56:59.682548 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.682554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.683376 (+ 822us) server_negotiation.cc:407] Connection header received
0925 14:56:59.683575 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.683580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.683653 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.683748 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:59.684291 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.685039 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.686709 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.687435 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.687596 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.687662 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.688088 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.688220 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:56:59.688501 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.682614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:34931 (local address 127.17.133.195:38839)
0925 14:56:59.683284 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:59.683297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:59.683433 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:59.683763 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:59.683767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:59.683787 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:59.684118 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.684134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.685195 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.685200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.686528 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:59.686538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.687611 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.687617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:59.687929 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.687959 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 14:56:59.688045 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:56:59.688146 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":22,"thread_start_us":328,"threads_started":1}
I20260925 14:56:59.688797 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a619ecb6f74d410094ee0f32d1513fac" candidate_uuid: "c618fc59c4ca4bb2b01593515479721f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2714c36fffb47f0a8be53ca291e41fd" is_pre_election: true
I20260925 14:56:59.689402 4374 raft_consensus.cc:2454] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c618fc59c4ca4bb2b01593515479721f in term 0.
I20260925 14:56:59.689604 4505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a619ecb6f74d410094ee0f32d1513fac" candidate_uuid: "c618fc59c4ca4bb2b01593515479721f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5213d009845340e68e57bce0d839dcb5" is_pre_election: true
I20260925 14:56:59.690101 4505 raft_consensus.cc:2454] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c618fc59c4ca4bb2b01593515479721f in term 0.
I20260925 14:56:59.690145 4571 leader_election.cc:302] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c618fc59c4ca4bb2b01593515479721f, e2714c36fffb47f0a8be53ca291e41fd; no voters:
I20260925 14:56:59.690514 4704 raft_consensus.cc:2781] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:59.690696 4704 raft_consensus.cc:482] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:59.690810 4704 raft_consensus.cc:3037] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:59.693007 4704 raft_consensus.cc:504] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } }
I20260925 14:56:59.693610 4704 leader_election.cc:288] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [CANDIDATE]: Term 1 election: Requested vote from peers 5213d009845340e68e57bce0d839dcb5 (127.17.133.194:34931), e2714c36fffb47f0a8be53ca291e41fd (127.17.133.193:36177)
I20260925 14:56:59.693939 4505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a619ecb6f74d410094ee0f32d1513fac" candidate_uuid: "c618fc59c4ca4bb2b01593515479721f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5213d009845340e68e57bce0d839dcb5"
I20260925 14:56:59.694151 4505 raft_consensus.cc:3037] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:59.694164 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a619ecb6f74d410094ee0f32d1513fac" candidate_uuid: "c618fc59c4ca4bb2b01593515479721f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2714c36fffb47f0a8be53ca291e41fd"
I20260925 14:56:59.694348 4374 raft_consensus.cc:3037] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:59.697405 4505 raft_consensus.cc:2454] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c618fc59c4ca4bb2b01593515479721f in term 1.
I20260925 14:56:59.697454 4374 raft_consensus.cc:2454] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c618fc59c4ca4bb2b01593515479721f in term 1.
I20260925 14:56:59.697856 4569 leader_election.cc:302] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5213d009845340e68e57bce0d839dcb5, c618fc59c4ca4bb2b01593515479721f; no voters:
I20260925 14:56:59.698179 4704 raft_consensus.cc:2781] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:59.698626 4704 raft_consensus.cc:686] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 1 LEADER]: Becoming Leader. State: Replica: c618fc59c4ca4bb2b01593515479721f, State: Running, Role: LEADER
I20260925 14:56:59.699080 4704 consensus_queue.cc:237] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } }
I20260925 14:56:59.703642 4233 catalog_manager.cc:5495] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f reported cstate change: term changed from 0 to 1, leader changed from <none> to c618fc59c4ca4bb2b01593515479721f (127.17.133.195). New cstate: current_term: 1 leader_uuid: "c618fc59c4ca4bb2b01593515479721f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c618fc59c4ca4bb2b01593515479721f" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 37677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 } health_report { overall_health: UNKNOWN } } }
I20260925 14:56:59.739276 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.734090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.17.133.195:37677)
0925 14:56:59.734283 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.734287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.734512 (+ 225us) server_negotiation.cc:407] Connection header received
0925 14:56:59.734672 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.734677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.734754 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.734875 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:59.735211 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.735922 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.737228 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.737641 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.737735 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.738175 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:59.738682 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:59.738804 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.738933 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.739000 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 14:56:59.742193 4681 tablet.cc:2369] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:59.747538 4505 raft_consensus.cc:1261] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [term 1 FOLLOWER]: Refusing update from remote peer c618fc59c4ca4bb2b01593515479721f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:59.747537 4374 raft_consensus.cc:1261] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [term 1 FOLLOWER]: Refusing update from remote peer c618fc59c4ca4bb2b01593515479721f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:59.748505 4704 consensus_queue.cc:1035] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5213d009845340e68e57bce0d839dcb5" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:59.748852 4708 consensus_queue.cc:1035] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2714c36fffb47f0a8be53ca291e41fd" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:59.758773 4711 mvcc.cc:205] Tried to move back new op lower bound from 7333266308076027904 to 7333266307891032064. Current Snapshot: MvccSnapshot[applied={T|T < 7333266308076027904}]
I20260925 14:56:59.774813 4231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:59.775118 4231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:59.779002 4231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 560198a9b8cd4bbfb973294b8741233c: Sending DeleteTablet for 3 replicas of tablet a619ecb6f74d410094ee0f32d1513fac
I20260925 14:56:59.780071 4485 tablet_service.cc:1514] Processing DeleteTablet for tablet a619ecb6f74d410094ee0f32d1513fac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:59 UTC) from {username='slave'} at 127.0.0.1:59936
I20260925 14:56:59.780126 4615 tablet_service.cc:1514] Processing DeleteTablet for tablet a619ecb6f74d410094ee0f32d1513fac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:59 UTC) from {username='slave'} at 127.0.0.1:41108
I20260925 14:56:59.780241 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet a619ecb6f74d410094ee0f32d1513fac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:59 UTC) from {username='slave'} at 127.0.0.1:42304
I20260925 14:56:59.782696 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 4292
I20260925 14:56:59.791792 4723 tablet_replica.cc:330] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: stopping tablet replica
I20260925 14:56:59.792354 4723 raft_consensus.cc:2229] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:59.792999 4723 raft_consensus.cc:2258] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:59.795537 4723 ts_tablet_manager.cc:1905] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:59.801049 4722 tablet_replica.cc:330] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: stopping tablet replica
I20260925 14:56:59.801404 4722 raft_consensus.cc:2229] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:59.801851 4722 raft_consensus.cc:2258] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:59.802567 4721 tablet_replica.cc:330] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: stopping tablet replica
I20260925 14:56:59.805590 4722 ts_tablet_manager.cc:1905] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:59.806286 4723 ts_tablet_manager.cc:1918] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:59.806464 4723 log.cc:1196] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/a619ecb6f74d410094ee0f32d1513fac
I20260925 14:56:59.807006 4723 ts_tablet_manager.cc:1939] T a619ecb6f74d410094ee0f32d1513fac P e2714c36fffb47f0a8be53ca291e41fd: Deleting consensus metadata
I20260925 14:56:59.808871 4220 catalog_manager.cc:4841] TS e2714c36fffb47f0a8be53ca291e41fd (127.17.133.193:36177): tablet a619ecb6f74d410094ee0f32d1513fac (table test-table [id=20d67f96bf25451c95baf8315d13e871]) successfully deleted
I20260925 14:56:59.809561 4721 raft_consensus.cc:2229] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:59.810223 4721 raft_consensus.cc:2258] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:59.811443 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.784173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.17.133.193:36177)
0925 14:56:59.784393 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.784402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.799455 (+ 15053us) server_negotiation.cc:407] Connection header received
0925 14:56:59.802636 (+ 3181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.802642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.802747 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.802863 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:59.804115 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.805315 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.807481 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.808132 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.808231 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.809741 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:59.809754 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:59.809773 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:59.810511 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:59.810577 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:59.810602 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:59.810681 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.810973 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.811082 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:56:59.812121 4414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:56:59.813019 4721 ts_tablet_manager.cc:1905] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:59.816800 4722 ts_tablet_manager.cc:1918] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:59.816985 4722 log.cc:1196] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/a619ecb6f74d410094ee0f32d1513fac
I20260925 14:56:59.817551 4722 ts_tablet_manager.cc:1939] T a619ecb6f74d410094ee0f32d1513fac P 5213d009845340e68e57bce0d839dcb5: Deleting consensus metadata
I20260925 14:56:59.819536 4218 catalog_manager.cc:4841] TS 5213d009845340e68e57bce0d839dcb5 (127.17.133.194:34931): tablet a619ecb6f74d410094ee0f32d1513fac (table test-table [id=20d67f96bf25451c95baf8315d13e871]) successfully deleted
I20260925 14:56:59.820283 4721 ts_tablet_manager.cc:1918] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:59.820448 4721 log.cc:1196] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/a619ecb6f74d410094ee0f32d1513fac
I20260925 14:56:59.821002 4721 ts_tablet_manager.cc:1939] T a619ecb6f74d410094ee0f32d1513fac P c618fc59c4ca4bb2b01593515479721f: Deleting consensus metadata
I20260925 14:56:59.822749 4217 catalog_manager.cc:4841] TS c618fc59c4ca4bb2b01593515479721f (127.17.133.195:37677): tablet a619ecb6f74d410094ee0f32d1513fac (table test-table [id=20d67f96bf25451c95baf8315d13e871]) successfully deleted
I20260925 14:56:59.902566 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 4292
I20260925 14:56:59.920523 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 4423
I20260925 14:56:59.928529 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:59.921438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.17.133.194:34931)
0925 14:56:59.921656 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:59.921663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:59.921712 (+ 49us) server_negotiation.cc:407] Connection header received
0925 14:56:59.921780 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:59.921786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:59.921893 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:59.922006 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:59.922347 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.923478 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.925071 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:59.925706 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:59.925829 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:59.926867 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:59.926881 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:59.926901 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:56:59.927701 (+ 800us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:59.927763 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:56:59.927789 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:59.927909 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:59.928039 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:56:59.928158 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:56:59.929409 4545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:00.018206 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 4423
I20260925 14:57:00.041287 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 4553
I20260925 14:57:00.050537 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:00.042126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.17.133.195:37677)
0925 14:57:00.042331 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:00.042354 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:00.042455 (+ 101us) server_negotiation.cc:407] Connection header received
0925 14:57:00.042653 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:00.042659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:00.042755 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:00.042863 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:00.043371 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:00.044552 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:00.046352 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:00.047011 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:00.047150 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:00.048427 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:00.048438 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:00.048457 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:00.049278 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:00.049347 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:00.049370 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:00.049487 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:00.050018 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 14:57:00.050132 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:57:00.051405 4675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:00.171914 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 4553
I20260925 14:57:00.192101 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 4202
I20260925 14:57:00.193228 4263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:00.302354 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 4202
2026-09-25T14:57:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2784 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:00 dist-test-slave-v5vl krb5kdc[4731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:00 dist-test-slave-v5vl krb5kdc[4731](info): set up 2 sockets
Sep 25 14:57:00 dist-test-slave-v5vl krb5kdc[4731](info): commencing operation
krb5kdc: starting...
W20260925 14:57:01.404771 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:01 dist-test-slave-v5vl krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:57:01.660279 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:38107
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:38107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:01.985045 4744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:01.985327 4744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:01.985416 4744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:01.992384 4744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:01.992533 4744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:01.992648 4744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:01.992717 4744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:01.992761 4744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:02.002583 4744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:38107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:02.003891 4744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:02.005123 4744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:02.016884 4752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:02.016901 4750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:02.016880 4749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:02.017756 4744 server_base.cc:1034] running on GCE node
I20260925 14:57:02.018465 4744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:02.019722 4744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:02.020915 4744 hybrid_clock.cc:648] HybridClock initialized: now 1790348222020864 us; error 39 us; skew 500 ppm
I20260925 14:57:02.021242 4744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:02 dist-test-slave-v5vl krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348222, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:02.024305 4744 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:57:02.025854 4744 webserver.cc:460] Webserver started at http://127.17.133.254:42799/ using document root <none> and password file <none>
I20260925 14:57:02.026517 4744 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:02.026660 4744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:02.026971 4744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:02.029973 4744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "12ccc7fcaedf41618ba13fc4bf362868"
format_stamp: "Formatted at 2026-09-25 14:57:02 on dist-test-slave-v5vl"
server_key: "fb5c30d76f5dae02b0f2c01a5d631d6c"
server_key_iv: "063ff123f3abcb2c727c1e0d028e7035"
server_key_version: "encryptionkey@0"
I20260925 14:57:02.030738 4744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "12ccc7fcaedf41618ba13fc4bf362868"
format_stamp: "Formatted at 2026-09-25 14:57:02 on dist-test-slave-v5vl"
server_key: "fb5c30d76f5dae02b0f2c01a5d631d6c"
server_key_iv: "063ff123f3abcb2c727c1e0d028e7035"
server_key_version: "encryptionkey@0"
I20260925 14:57:02.035878 4744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 14:57:02.039357 4759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:02.040760 4744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:02.040931 4744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "12ccc7fcaedf41618ba13fc4bf362868"
format_stamp: "Formatted at 2026-09-25 14:57:02 on dist-test-slave-v5vl"
server_key: "fb5c30d76f5dae02b0f2c01a5d631d6c"
server_key_iv: "063ff123f3abcb2c727c1e0d028e7035"
server_key_version: "encryptionkey@0"
I20260925 14:57:02.041134 4744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:02.055729 4744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:02.059911 4744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:02.060189 4744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:02.078141 4744 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:38107
I20260925 14:57:02.078161 4810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:38107 every 8 connection(s)
I20260925 14:57:02.079542 4744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:02.083246 4811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:02.084311 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 4744
I20260925 14:57:02.084498 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:02.084957 17943 external_mini_cluster.cc:1428] Setting key d1761afd457784289ad8ea3077493746
I20260925 14:57:02.096848 4811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868: Bootstrap starting.
Sep 25 14:57:02 dist-test-slave-v5vl krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:02.100474 4811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:02.101552 4811 log.cc:824] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:02.104184 4811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868: No bootstrap required, opened a new log
I20260925 14:57:02.107465 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:02.086722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.17.133.254:38107)
0925 14:57:02.087530 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:02.087541 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:02.087587 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:57:02.089281 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:02.089303 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:02.089682 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:02.089990 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:02.091770 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:02.093060 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:02.095033 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:02.095667 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:02.095802 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:02.100415 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:02.100422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:02.100436 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:02.100472 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:02.103522 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:02.104206 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:02.104212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:02.104217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:02.104418 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:02.104847 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:02.104852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:02.104856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:02.105352 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:02.105662 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:02.106093 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 14:57:02.106504 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":367,"threads_started":1}
I20260925 14:57:02.112258 4811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ccc7fcaedf41618ba13fc4bf362868" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 38107 } }
I20260925 14:57:02.112666 4811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:02.112774 4811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ccc7fcaedf41618ba13fc4bf362868, State: Initialized, Role: FOLLOWER
I20260925 14:57:02.113373 4811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ccc7fcaedf41618ba13fc4bf362868" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 38107 } }
I20260925 14:57:02.113627 4811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:02.113768 4811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:02.113955 4811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:02.116005 4811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ccc7fcaedf41618ba13fc4bf362868" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 38107 } }
I20260925 14:57:02.116463 4811 leader_election.cc:302] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12ccc7fcaedf41618ba13fc4bf362868; no voters:
I20260925 14:57:02.117156 4811 leader_election.cc:288] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:02.117303 4816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:02.117956 4816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [term 1 LEADER]: Becoming Leader. State: Replica: 12ccc7fcaedf41618ba13fc4bf362868, State: Running, Role: LEADER
I20260925 14:57:02.118505 4816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ccc7fcaedf41618ba13fc4bf362868" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 38107 } }
I20260925 14:57:02.119376 4811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:02.121596 4817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12ccc7fcaedf41618ba13fc4bf362868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ccc7fcaedf41618ba13fc4bf362868" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 38107 } } }
I20260925 14:57:02.121642 4818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12ccc7fcaedf41618ba13fc4bf362868. Latest consensus state: current_term: 1 leader_uuid: "12ccc7fcaedf41618ba13fc4bf362868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ccc7fcaedf41618ba13fc4bf362868" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 38107 } } }
I20260925 14:57:02.121961 4817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:02.122028 4818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:02.127719 4821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:02.132786 4821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:02.140872 4821 catalog_manager.cc:1347] Generated new cluster ID: 0b2a89c153824b27ad44152e1a9467f1
I20260925 14:57:02.140973 4821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:02.156558 4821 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:02.158497 4821 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:02.169512 4821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868: Generated new TSK 0
I20260925 14:57:02.170622 4821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.133.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
Entry for principal kudu/127.17.133.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab.
I20260925 14:57:02.237439 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:38107
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:02.563819 4837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:02.564097 4837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:02.564278 4837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:02.571655 4837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:02.571753 4837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:02.571900 4837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:57:02.581478 4837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.193.keytab
--principal=kudu/127.17.133.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:38107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:02.582751 4837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:02.584055 4837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:02.597239 4842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:02.597275 4843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:02.597311 4845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:02.598218 4837 server_base.cc:1034] running on GCE node
I20260925 14:57:02.598770 4837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:02.599794 4837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:02.601015 4837 hybrid_clock.cc:648] HybridClock initialized: now 1790348222600947 us; error 55 us; skew 500 ppm
I20260925 14:57:02.601346 4837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:02 dist-test-slave-v5vl krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348222, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:02.604373 4837 init.cc:377] Logged in from keytab as kudu/127.17.133.193@KRBTEST.COM (short username kudu)
I20260925 14:57:02.605895 4837 webserver.cc:460] Webserver started at http://127.17.133.193:34025/ using document root <none> and password file <none>
I20260925 14:57:02.606575 4837 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:02.606719 4837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:02.607033 4837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:02.610073 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "0aa3e7ca6da5446e871db83e2e266a67"
format_stamp: "Formatted at 2026-09-25 14:57:02 on dist-test-slave-v5vl"
server_key: "334906111266a7a81d86c070b7546a64"
server_key_iv: "079ec134501ea6b4a7856f5d3752a8d9"
server_key_version: "encryptionkey@0"
I20260925 14:57:02.610920 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "0aa3e7ca6da5446e871db83e2e266a67"
format_stamp: "Formatted at 2026-09-25 14:57:02 on dist-test-slave-v5vl"
server_key: "334906111266a7a81d86c070b7546a64"
server_key_iv: "079ec134501ea6b4a7856f5d3752a8d9"
server_key_version: "encryptionkey@0"
I20260925 14:57:02.616031 4837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 14:57:02.619500 4852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:02.620999 4837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 14:57:02.621152 4837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "0aa3e7ca6da5446e871db83e2e266a67"
format_stamp: "Formatted at 2026-09-25 14:57:02 on dist-test-slave-v5vl"
server_key: "334906111266a7a81d86c070b7546a64"
server_key_iv: "079ec134501ea6b4a7856f5d3752a8d9"
server_key_version: "encryptionkey@0"
I20260925 14:57:02.621352 4837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:02.635064 4837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:02.638785 4837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:02.639070 4837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:02.640028 4837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:02.641332 4837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:02.641438 4837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:02.641582 4837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:02.641664 4837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:02.674998 4837 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:35233
I20260925 14:57:02.675037 4964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:35233 every 8 connection(s)
I20260925 14:57:02.676295 4837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:02.680670 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 4837
I20260925 14:57:02.680855 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:02.681329 17943 external_mini_cluster.cc:1428] Setting key 19632c3b384c8d8237acea5a9d7e404e
Sep 25 14:57:02 dist-test-slave-v5vl krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348222, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.193@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:02.696444 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:02.678903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:32925 (local address 127.17.133.254:38107)
0925 14:57:02.679375 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:02.679379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:02.680178 (+ 799us) server_negotiation.cc:407] Connection header received
0925 14:57:02.682293 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:02.682298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:02.682404 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:02.682521 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:02.684102 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:02.684892 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:02.686468 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:02.686883 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:02.687016 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:02.690668 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:02.690672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:02.690675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:02.690705 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:02.693218 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:02.694145 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:02.694149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:02.694151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:02.694264 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:02.694773 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:02.694776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:02.694778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:02.695037 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:02.695173 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:02.696028 (+ 855us) server_negotiation.cc:299] Negotiation successful
0925 14:57:02.696203 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":254,"threads_started":1}
I20260925 14:57:02.698084 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:02.679327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:38107 (local address 127.17.133.193:32925)
0925 14:57:02.680024 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:02.680062 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:02.682007 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:02.682758 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:02.682767 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:02.683355 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:02.683932 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:02.683950 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:02.685048 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:02.685059 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:02.686319 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:02.686331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:02.687106 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:02.687116 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:02.687360 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:02.688033 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:02.688053 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:02.690444 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:02.693368 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:02.693379 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:02.693398 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:02.694027 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:02.694374 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:02.694382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:02.694388 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:02.694669 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:02.695179 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:02.695207 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:02.695637 (+ 430us) client_negotiation.cc:769] Sending connection context
0925 14:57:02.695957 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 14:57:02.696516 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":257,"threads_started":1}
I20260925 14:57:02.699647 4965 heartbeater.cc:347] Connected to a master server at 127.17.133.254:38107
I20260925 14:57:02.699947 4965 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:02.700699 4965 heartbeater.cc:511] Master 127.17.133.254:38107 requested a full tablet report, sending...
I20260925 14:57:02.702773 4776 ts_manager.cc:194] Registered new tserver with Master: 0aa3e7ca6da5446e871db83e2e266a67 (127.17.133.193:35233)
I20260925 14:57:02.704152 4776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.193@KRBTEST.COM'} at 127.17.133.193:32925
WARNING: no policy specified for kudu/127.17.133.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
Entry for principal kudu/127.17.133.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab.
I20260925 14:57:02.735576 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:38107
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:03.054066 4971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:03.054401 4971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:03.054570 4971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:03.061998 4971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:03.062146 4971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:03.062331 4971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:57:03.072048 4971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.194.keytab
--principal=kudu/127.17.133.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:38107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:03.073360 4971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:03.074625 4971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:03.088523 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:03.088594 4977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:03.088814 4976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:03.090094 4971 server_base.cc:1034] running on GCE node
I20260925 14:57:03.090757 4971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:03.091883 4971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:03.093123 4971 hybrid_clock.cc:648] HybridClock initialized: now 1790348223093052 us; error 55 us; skew 500 ppm
I20260925 14:57:03.093493 4971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:03 dist-test-slave-v5vl krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348223, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:03.101398 4971 init.cc:377] Logged in from keytab as kudu/127.17.133.194@KRBTEST.COM (short username kudu)
I20260925 14:57:03.102852 4971 webserver.cc:460] Webserver started at http://127.17.133.194:45029/ using document root <none> and password file <none>
I20260925 14:57:03.103515 4971 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:03.103657 4971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:03.103957 4971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:03.106828 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "72e1474e74b84f22bb6242a1347c99d4"
format_stamp: "Formatted at 2026-09-25 14:57:03 on dist-test-slave-v5vl"
server_key: "3b772659471f78c6cb41d871d0ed2d9b"
server_key_iv: "8e5d98b8470ae1b190e91edc4f705f08"
server_key_version: "encryptionkey@0"
I20260925 14:57:03.107585 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "72e1474e74b84f22bb6242a1347c99d4"
format_stamp: "Formatted at 2026-09-25 14:57:03 on dist-test-slave-v5vl"
server_key: "3b772659471f78c6cb41d871d0ed2d9b"
server_key_iv: "8e5d98b8470ae1b190e91edc4f705f08"
server_key_version: "encryptionkey@0"
I20260925 14:57:03.112571 4971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:57:03.115792 4986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:03.117063 4971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:57:03.117244 4971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "72e1474e74b84f22bb6242a1347c99d4"
format_stamp: "Formatted at 2026-09-25 14:57:03 on dist-test-slave-v5vl"
server_key: "3b772659471f78c6cb41d871d0ed2d9b"
server_key_iv: "8e5d98b8470ae1b190e91edc4f705f08"
server_key_version: "encryptionkey@0"
I20260925 14:57:03.117462 4971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:03.128841 4971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:03.132309 4971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:03.132630 4971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:03.133510 4971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:03.134711 4971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:03.134809 4971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:03.134955 4971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:03.135031 4971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:03.167177 4971 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:32797
I20260925 14:57:03.167228 5098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:32797 every 8 connection(s)
I20260925 14:57:03.168454 4971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:03.178692 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 4971
I20260925 14:57:03.178860 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:03.179287 17943 external_mini_cluster.cc:1428] Setting key 115d0c736d3552ece16bf25bfac707b1
Sep 25 14:57:03 dist-test-slave-v5vl krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348223, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.194@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:03.187855 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.171040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:57045 (local address 127.17.133.254:38107)
0925 14:57:03.171219 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:03.171224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:03.172285 (+ 1061us) server_negotiation.cc:407] Connection header received
0925 14:57:03.174111 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:03.174116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:03.174200 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:03.174320 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:03.175695 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.176418 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.177946 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.178376 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.178546 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.182207 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:03.182212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:03.182217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:03.182256 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:03.184882 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:03.185575 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:03.185580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:03.185584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:03.185717 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:03.186143 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:03.186148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:03.186151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:03.186413 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:03.186571 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:03.187271 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 14:57:03.187520 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:57:03.189180 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.171473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:38107 (local address 127.17.133.194:57045)
0925 14:57:03.172125 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:03.172162 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:03.173842 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:03.174552 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:03.174559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:03.175041 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:03.175542 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.175556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.176573 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.176579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.177788 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.177799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.178596 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.178604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.178886 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.179619 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:03.179644 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:03.182002 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:03.185017 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:03.185026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:03.185041 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:03.185448 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:03.185827 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:03.185833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:03.185836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:03.186026 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:03.186552 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:03.186560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:03.186893 (+ 333us) client_negotiation.cc:769] Sending connection context
0925 14:57:03.187176 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 14:57:03.187713 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":247,"threads_started":1}
I20260925 14:57:03.190604 5099 heartbeater.cc:347] Connected to a master server at 127.17.133.254:38107
I20260925 14:57:03.190908 5099 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:03.191708 5099 heartbeater.cc:511] Master 127.17.133.254:38107 requested a full tablet report, sending...
I20260925 14:57:03.193331 4776 ts_manager.cc:194] Registered new tserver with Master: 72e1474e74b84f22bb6242a1347c99d4 (127.17.133.194:32797)
I20260925 14:57:03.194561 4776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.194@KRBTEST.COM'} at 127.17.133.194:57045
WARNING: no policy specified for kudu/127.17.133.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
Entry for principal kudu/127.17.133.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab.
I20260925 14:57:03.235440 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:38107
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:03.559485 5104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:03.559770 5104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:03.559866 5104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:03.567377 5104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:03.567497 5104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:03.567608 5104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:57:03.577548 5104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.195.keytab
--principal=kudu/127.17.133.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:38107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:03.578820 5104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:03.580066 5104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:03.593067 5109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:03.593015 5110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:03.593099 5112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:03.594105 5104 server_base.cc:1034] running on GCE node
I20260925 14:57:03.594717 5104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:03.595671 5104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:03.596903 5104 hybrid_clock.cc:648] HybridClock initialized: now 1790348223596859 us; error 53 us; skew 500 ppm
I20260925 14:57:03.597384 5104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:03 dist-test-slave-v5vl krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348223, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:03.601878 5104 init.cc:377] Logged in from keytab as kudu/127.17.133.195@KRBTEST.COM (short username kudu)
I20260925 14:57:03.603363 5104 webserver.cc:460] Webserver started at http://127.17.133.195:36379/ using document root <none> and password file <none>
I20260925 14:57:03.604074 5104 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:03.604220 5104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:03.604547 5104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:03.607769 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "e3351a801d0c48b4aede1408ba3893d3"
format_stamp: "Formatted at 2026-09-25 14:57:03 on dist-test-slave-v5vl"
server_key: "b6ca8d1ab439123940cd2d87769aceca"
server_key_iv: "ec41394d73400955907c84a310676804"
server_key_version: "encryptionkey@0"
I20260925 14:57:03.608553 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "e3351a801d0c48b4aede1408ba3893d3"
format_stamp: "Formatted at 2026-09-25 14:57:03 on dist-test-slave-v5vl"
server_key: "b6ca8d1ab439123940cd2d87769aceca"
server_key_iv: "ec41394d73400955907c84a310676804"
server_key_version: "encryptionkey@0"
I20260925 14:57:03.614074 5104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 14:57:03.617719 5119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:03.619125 5104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:57:03.619310 5104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "e3351a801d0c48b4aede1408ba3893d3"
format_stamp: "Formatted at 2026-09-25 14:57:03 on dist-test-slave-v5vl"
server_key: "b6ca8d1ab439123940cd2d87769aceca"
server_key_iv: "ec41394d73400955907c84a310676804"
server_key_version: "encryptionkey@0"
I20260925 14:57:03.619529 5104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:03.663882 5104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:03.668303 5104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:03.668656 5104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:03.669574 5104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:03.670845 5104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:03.670914 5104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:03.670996 5104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:03.671039 5104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:03.704039 5104 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:43117
I20260925 14:57:03.704087 5231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:43117 every 8 connection(s)
I20260925 14:57:03.705463 5104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:03.706949 4965 heartbeater.cc:503] Master 127.17.133.254:38107 was elected leader, sending a full tablet report...
I20260925 14:57:03.709365 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 5104
I20260925 14:57:03.709547 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:03.710114 17943 external_mini_cluster.cc:1428] Setting key 9ce0a7309e1338136ae707ad5cb0e4e0
Sep 25 14:57:03 dist-test-slave-v5vl krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348223, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.195@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:03.725859 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.707976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:48229 (local address 127.17.133.254:38107)
0925 14:57:03.708416 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:03.708422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:03.709715 (+ 1293us) server_negotiation.cc:407] Connection header received
0925 14:57:03.712024 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:03.712028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:03.712102 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:03.712180 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:03.713880 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.714638 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.716076 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.716452 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.716538 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.719825 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:03.719831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:03.719834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:03.719865 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:03.722572 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:03.723303 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:03.723307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:03.723309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:03.723461 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:03.723987 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:03.723991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:03.724012 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:03.724380 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:03.724507 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:03.725371 (+ 864us) server_negotiation.cc:299] Negotiation successful
0925 14:57:03.725585 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":223,"threads_started":1}
I20260925 14:57:03.727005 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.708629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:38107 (local address 127.17.133.195:48229)
0925 14:57:03.709549 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:03.709611 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:03.711748 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:03.712450 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:03.712462 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:03.713171 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:03.713731 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.713748 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.714753 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.714757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.715954 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.715966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.716683 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.716689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.716880 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.717466 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:03.717492 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:03.719581 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:03.722688 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:03.722695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:03.722709 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:03.723202 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:03.723597 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:03.723602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:03.723606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:03.723826 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:03.724525 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:03.724534 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:03.724967 (+ 433us) client_negotiation.cc:769] Sending connection context
0925 14:57:03.725243 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 14:57:03.725733 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":332,"threads_started":1}
I20260925 14:57:03.728264 5232 heartbeater.cc:347] Connected to a master server at 127.17.133.254:38107
I20260925 14:57:03.728515 5232 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:03.729210 5232 heartbeater.cc:511] Master 127.17.133.254:38107 requested a full tablet report, sending...
I20260925 14:57:03.730491 4776 ts_manager.cc:194] Registered new tserver with Master: e3351a801d0c48b4aede1408ba3893d3 (127.17.133.195:43117)
I20260925 14:57:03.731269 4776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.133.195@KRBTEST.COM'} at 127.17.133.195:48229
I20260925 14:57:03.737643 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:03.753495 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.741682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.17.133.254:38107)
0925 14:57:03.741887 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:03.741896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:03.742270 (+ 374us) server_negotiation.cc:407] Connection header received
0925 14:57:03.742393 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:03.742396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:03.742482 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:03.742564 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:03.743881 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.744651 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.746009 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.746392 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.746490 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.748726 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:03.748730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:03.748733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:03.748763 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:03.751123 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:03.751698 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:03.751702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:03.751704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:03.751816 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:03.752126 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:03.752129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:03.752131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:03.752433 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:03.752700 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:03.752977 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 14:57:03.753150 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:57:03.758203 4776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:03.766639 4776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:03.802431 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.778987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.17.133.195:43117)
0925 14:57:03.791482 (+ 12495us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:03.791497 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:03.793269 (+ 1772us) server_negotiation.cc:407] Connection header received
0925 14:57:03.794226 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:03.794232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:03.794523 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:03.794695 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:03.795800 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.797386 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.799702 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.801198 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.801402 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.801600 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:03.801881 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 14:57:03.802052 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12208,"thread_start_us":12536,"threads_started":1}
I20260925 14:57:03.802543 5245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.778726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:43117 (local address 127.0.0.1:43404)
0925 14:57:03.793130 (+ 14404us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:03.793147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:03.793906 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:03.795205 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:03.795209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:03.795234 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:03.795583 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.795595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.797537 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.797542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.799530 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.799551 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.801385 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.801391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.801640 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.801699 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 14:57:03.801848 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 14:57:03.802076 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14266,"mutex_wait_us":54,"thread_start_us":13221,"threads_started":1}
I20260925 14:57:03.802574 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.778273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:35233 (local address 127.0.0.1:41874)
0925 14:57:03.792494 (+ 14221us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:03.792551 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:03.792854 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:03.794461 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:03.794488 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:03.794537 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:03.795069 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.795083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.797266 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.797272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.799399 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.799423 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.801551 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.801559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.801749 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.801784 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:57:03.801855 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:57:03.802078 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13859,"mutex_wait_us":65,"thread_start_us":15020,"threads_started":1}
I20260925 14:57:03.803768 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.778472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.17.133.193:35233)
0925 14:57:03.793642 (+ 15170us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:03.793652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:03.793679 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:57:03.793790 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:03.793801 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:03.794102 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:03.795265 (+ 1163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:03.795489 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.797118 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.799602 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.801307 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.803039 (+ 1732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.803200 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:03.803262 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 14:57:03.803432 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14998,"thread_start_us":14822,"threads_started":1}
I20260925 14:57:03.803833 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.793670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:32797 (local address 127.0.0.1:44060)
0925 14:57:03.794740 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:03.794761 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:03.794866 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:03.795966 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:03.795971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:03.795994 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:03.796407 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.796419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.798748 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.798754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.800949 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:03.800964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.803096 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.803102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:03.803258 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.803287 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 14:57:03.803379 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 14:57:03.803487 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":31,"thread_start_us":1261,"threads_started":1}
I20260925 14:57:03.804276 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:03.793957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.17.133.194:32797)
0925 14:57:03.795360 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:03.795368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:03.795394 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:57:03.795499 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:03.795507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:03.795723 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:03.796578 (+ 855us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:03.796995 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.798582 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.801128 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:03.802573 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:03.802739 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:03.802958 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:03.803438 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 14:57:03.803601 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":1097,"threads_started":1}
I20260925 14:57:03.806795 5167 tablet_service.cc:1467] Processing CreateTablet for tablet d8e9abda4fa842228b99a66d90d15205 (DEFAULT_TABLE table=test-table [id=bd06fc91ce5d49c9b143a6c3a542820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:03.807318 4900 tablet_service.cc:1467] Processing CreateTablet for tablet d8e9abda4fa842228b99a66d90d15205 (DEFAULT_TABLE table=test-table [id=bd06fc91ce5d49c9b143a6c3a542820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:03.808156 5167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e9abda4fa842228b99a66d90d15205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:03.808072 5034 tablet_service.cc:1467] Processing CreateTablet for tablet d8e9abda4fa842228b99a66d90d15205 (DEFAULT_TABLE table=test-table [id=bd06fc91ce5d49c9b143a6c3a542820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:03.808750 4900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e9abda4fa842228b99a66d90d15205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:03.809717 5034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e9abda4fa842228b99a66d90d15205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:03.819322 5251 tablet_bootstrap.cc:492] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: Bootstrap starting.
I20260925 14:57:03.823760 5251 tablet_bootstrap.cc:654] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:03.825309 5251 log.cc:824] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:03.825433 5253 tablet_bootstrap.cc:492] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: Bootstrap starting.
I20260925 14:57:03.827272 5252 tablet_bootstrap.cc:492] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: Bootstrap starting.
I20260925 14:57:03.829070 5251 tablet_bootstrap.cc:492] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: No bootstrap required, opened a new log
I20260925 14:57:03.829247 5253 tablet_bootstrap.cc:654] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:03.829360 5251 ts_tablet_manager.cc:1397] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 14:57:03.830479 5253 log.cc:824] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:03.830803 5252 tablet_bootstrap.cc:654] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:03.832194 5252 log.cc:824] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:03.833773 5253 tablet_bootstrap.cc:492] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: No bootstrap required, opened a new log
I20260925 14:57:03.834039 5253 ts_tablet_manager.cc:1397] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 14:57:03.835155 5252 tablet_bootstrap.cc:492] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: No bootstrap required, opened a new log
I20260925 14:57:03.835413 5252 ts_tablet_manager.cc:1397] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260925 14:57:03.837734 5251 raft_consensus.cc:348] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } }
I20260925 14:57:03.838091 5251 raft_consensus.cc:374] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:03.838213 5251 raft_consensus.cc:729] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3351a801d0c48b4aede1408ba3893d3, State: Initialized, Role: FOLLOWER
I20260925 14:57:03.838871 5251 consensus_queue.cc:260] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } }
I20260925 14:57:03.840152 5232 heartbeater.cc:503] Master 127.17.133.254:38107 was elected leader, sending a full tablet report...
I20260925 14:57:03.840660 5251 ts_tablet_manager.cc:1428] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260925 14:57:03.844960 5253 raft_consensus.cc:348] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } }
I20260925 14:57:03.845386 5253 raft_consensus.cc:374] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:03.845506 5253 raft_consensus.cc:729] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e1474e74b84f22bb6242a1347c99d4, State: Initialized, Role: FOLLOWER
I20260925 14:57:03.845600 5252 raft_consensus.cc:348] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } }
I20260925 14:57:03.846071 5252 raft_consensus.cc:374] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:03.846191 5252 raft_consensus.cc:729] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa3e7ca6da5446e871db83e2e266a67, State: Initialized, Role: FOLLOWER
I20260925 14:57:03.846149 5253 consensus_queue.cc:260] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } }
I20260925 14:57:03.846915 5252 consensus_queue.cc:260] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } }
I20260925 14:57:03.847314 5099 heartbeater.cc:503] Master 127.17.133.254:38107 was elected leader, sending a full tablet report...
I20260925 14:57:03.847949 5253 ts_tablet_manager.cc:1428] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260925 14:57:03.848623 5252 ts_tablet_manager.cc:1428] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260925 14:57:03.920742 5100 tablet.cc:2369] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:03.928712 4966 tablet.cc:2369] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:03.957146 5233 tablet.cc:2369] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:04.153769 5259 raft_consensus.cc:482] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:04.154085 5259 raft_consensus.cc:504] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } }
I20260925 14:57:04.155859 5259 leader_election.cc:288] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3351a801d0c48b4aede1408ba3893d3 (127.17.133.195:43117), 72e1474e74b84f22bb6242a1347c99d4 (127.17.133.194:32797)
I20260925 14:57:04.163445 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:04.156814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:51813 (local address 127.17.133.194:32797)
0925 14:57:04.157291 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:04.157300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:04.158037 (+ 737us) server_negotiation.cc:407] Connection header received
0925 14:57:04.158250 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:04.158254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:04.158312 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:04.158423 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:04.159058 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.159781 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.161689 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.162435 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.162622 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:04.162701 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:04.163051 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 14:57:04.163175 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298}
I20260925 14:57:04.163484 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:04.156530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:49463 (local address 127.17.133.195:43117)
0925 14:57:04.157081 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:04.157090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:04.158077 (+ 987us) server_negotiation.cc:407] Connection header received
0925 14:57:04.158335 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:04.158339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:04.158394 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:04.158489 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:04.159269 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.159940 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.161743 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.162498 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.162677 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:04.162756 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:04.163098 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 14:57:04.163223 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260925 14:57:04.163652 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:04.156226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:32797 (local address 127.17.133.193:51813)
0925 14:57:04.157910 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:04.157930 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:04.158104 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:04.158437 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:04.158442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:04.158467 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:04.158909 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:04.158919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.159910 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.159918 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:04.161531 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:04.161543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.162668 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.162674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:04.162850 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:04.162914 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 14:57:04.163026 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:57:04.163157 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1554,"mutex_wait_us":77,"thread_start_us":1419,"threads_started":1}
I20260925 14:57:04.163949 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:04.156258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:43117 (local address 127.17.133.193:49463)
0925 14:57:04.157950 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:04.157965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:04.158177 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:04.158501 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:04.158505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:04.158532 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:04.159101 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:04.159110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.160084 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.160094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:04.161589 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:04.161601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.162724 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.162730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:04.162942 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:04.162965 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:57:04.163065 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 14:57:04.163188 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1585,"mutex_wait_us":36,"spinlock_wait_cycles":24832,"thread_start_us":1300,"threads_started":1}
I20260925 14:57:04.164798 5054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e9abda4fa842228b99a66d90d15205" candidate_uuid: "0aa3e7ca6da5446e871db83e2e266a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e1474e74b84f22bb6242a1347c99d4" is_pre_election: true
I20260925 14:57:04.164932 5187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e9abda4fa842228b99a66d90d15205" candidate_uuid: "0aa3e7ca6da5446e871db83e2e266a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3351a801d0c48b4aede1408ba3893d3" is_pre_election: true
I20260925 14:57:04.165333 5054 raft_consensus.cc:2454] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa3e7ca6da5446e871db83e2e266a67 in term 0.
I20260925 14:57:04.165377 5187 raft_consensus.cc:2454] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa3e7ca6da5446e871db83e2e266a67 in term 0.
I20260925 14:57:04.166026 4856 leader_election.cc:302] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0aa3e7ca6da5446e871db83e2e266a67, 72e1474e74b84f22bb6242a1347c99d4, e3351a801d0c48b4aede1408ba3893d3; no voters:
I20260925 14:57:04.166443 5259 raft_consensus.cc:2781] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:04.166596 5259 raft_consensus.cc:482] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:04.166724 5259 raft_consensus.cc:3037] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:04.169024 5259 raft_consensus.cc:504] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } }
I20260925 14:57:04.169771 5259 leader_election.cc:288] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [CANDIDATE]: Term 1 election: Requested vote from peers e3351a801d0c48b4aede1408ba3893d3 (127.17.133.195:43117), 72e1474e74b84f22bb6242a1347c99d4 (127.17.133.194:32797)
I20260925 14:57:04.170125 5187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e9abda4fa842228b99a66d90d15205" candidate_uuid: "0aa3e7ca6da5446e871db83e2e266a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3351a801d0c48b4aede1408ba3893d3"
I20260925 14:57:04.170332 5187 raft_consensus.cc:3037] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:04.170533 5054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e9abda4fa842228b99a66d90d15205" candidate_uuid: "0aa3e7ca6da5446e871db83e2e266a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e1474e74b84f22bb6242a1347c99d4"
I20260925 14:57:04.170733 5054 raft_consensus.cc:3037] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:04.173385 5187 raft_consensus.cc:2454] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa3e7ca6da5446e871db83e2e266a67 in term 1.
I20260925 14:57:04.173863 4856 leader_election.cc:302] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa3e7ca6da5446e871db83e2e266a67, e3351a801d0c48b4aede1408ba3893d3; no voters:
I20260925 14:57:04.174218 5259 raft_consensus.cc:2781] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:04.174919 5054 raft_consensus.cc:2454] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa3e7ca6da5446e871db83e2e266a67 in term 1.
I20260925 14:57:04.175699 5259 raft_consensus.cc:686] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 1 LEADER]: Becoming Leader. State: Replica: 0aa3e7ca6da5446e871db83e2e266a67, State: Running, Role: LEADER
I20260925 14:57:04.176326 5259 consensus_queue.cc:237] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } }
I20260925 14:57:04.181838 4776 catalog_manager.cc:5495] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aa3e7ca6da5446e871db83e2e266a67 (127.17.133.193). New cstate: current_term: 1 leader_uuid: "0aa3e7ca6da5446e871db83e2e266a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa3e7ca6da5446e871db83e2e266a67" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 } health_report { overall_health: UNKNOWN } } }
I20260925 14:57:04.256748 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:04.251853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.17.133.193:35233)
0925 14:57:04.252042 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:04.252047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:04.252064 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:57:04.252178 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:04.252182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:04.252257 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:04.252371 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:04.252694 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.253381 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.254868 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.255242 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.255336 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:04.255769 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:04.256244 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:04.256343 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:04.256430 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 14:57:04.256489 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:57:04.264449 5187 raft_consensus.cc:1261] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [term 1 FOLLOWER]: Refusing update from remote peer 0aa3e7ca6da5446e871db83e2e266a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:04.264448 5054 raft_consensus.cc:1261] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [term 1 FOLLOWER]: Refusing update from remote peer 0aa3e7ca6da5446e871db83e2e266a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:04.265374 5262 consensus_queue.cc:1035] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3351a801d0c48b4aede1408ba3893d3" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:04.265743 5259 consensus_queue.cc:1035] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e1474e74b84f22bb6242a1347c99d4" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:04.277179 5265 mvcc.cc:205] Tried to move back new op lower bound from 7333266326578151424 to 7333266326231629824. Current Snapshot: MvccSnapshot[applied={T|T < 7333266326578151424}]
I20260925 14:57:04.290315 4775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:04.290642 4775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:04.294529 4775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12ccc7fcaedf41618ba13fc4bf362868: Sending DeleteTablet for 3 replicas of tablet d8e9abda4fa842228b99a66d90d15205
I20260925 14:57:04.295694 5167 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e9abda4fa842228b99a66d90d15205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:04 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:43404
I20260925 14:57:04.295814 4900 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e9abda4fa842228b99a66d90d15205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:04 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:41874
I20260925 14:57:04.296008 5034 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e9abda4fa842228b99a66d90d15205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:04 UTC) from {username='kudu', principal='kudu/127.17.133.254@KRBTEST.COM'} at 127.0.0.1:44060
I20260925 14:57:04.297629 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 4837
I20260925 14:57:04.306424 5277 tablet_replica.cc:330] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: stopping tablet replica
I20260925 14:57:04.307029 5277 raft_consensus.cc:2229] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:04.307839 5277 raft_consensus.cc:2258] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:04.310212 5277 ts_tablet_manager.cc:1905] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:04.310843 5276 tablet_replica.cc:330] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: stopping tablet replica
I20260925 14:57:04.311295 5276 raft_consensus.cc:2229] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:57:04.318583 5277 ts_tablet_manager.cc:1918] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:04.319440 5276 raft_consensus.cc:2258] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:04.322391 5277 log.cc:1196] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/d8e9abda4fa842228b99a66d90d15205
I20260925 14:57:04.322602 5275 tablet_replica.cc:330] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: stopping tablet replica
I20260925 14:57:04.323082 5277 ts_tablet_manager.cc:1939] T d8e9abda4fa842228b99a66d90d15205 P 72e1474e74b84f22bb6242a1347c99d4: Deleting consensus metadata
I20260925 14:57:04.323251 5276 ts_tablet_manager.cc:1905] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:04.324643 5275 raft_consensus.cc:2229] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:04.325296 4760 catalog_manager.cc:4841] TS 72e1474e74b84f22bb6242a1347c99d4 (127.17.133.194:32797): tablet d8e9abda4fa842228b99a66d90d15205 (table test-table [id=bd06fc91ce5d49c9b143a6c3a542820c]) successfully deleted
I20260925 14:57:04.325384 5275 raft_consensus.cc:2258] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 14:57:04 dist-test-slave-v5vl krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.193@KRBTEST.COM
I20260925 14:57:04.327901 5275 ts_tablet_manager.cc:1905] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:04.332995 5276 ts_tablet_manager.cc:1918] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:04.333178 5276 log.cc:1196] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-0/wal/wals/d8e9abda4fa842228b99a66d90d15205
I20260925 14:57:04.333789 5276 ts_tablet_manager.cc:1939] T d8e9abda4fa842228b99a66d90d15205 P 0aa3e7ca6da5446e871db83e2e266a67: Deleting consensus metadata
I20260925 14:57:04.335696 4760 catalog_manager.cc:4841] TS 0aa3e7ca6da5446e871db83e2e266a67 (127.17.133.193:35233): tablet d8e9abda4fa842228b99a66d90d15205 (table test-table [id=bd06fc91ce5d49c9b143a6c3a542820c]) successfully deleted
I20260925 14:57:04.336138 5275 ts_tablet_manager.cc:1918] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:04.336304 5275 log.cc:1196] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/d8e9abda4fa842228b99a66d90d15205
I20260925 14:57:04.336851 5275 ts_tablet_manager.cc:1939] T d8e9abda4fa842228b99a66d90d15205 P e3351a801d0c48b4aede1408ba3893d3: Deleting consensus metadata
I20260925 14:57:04.336872 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:04.303976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.17.133.193:35233)
0925 14:57:04.304211 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:04.304218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:04.314627 (+ 10409us) server_negotiation.cc:407] Connection header received
0925 14:57:04.315214 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:04.315223 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:04.315368 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:04.315488 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:04.317868 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.319576 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.322310 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.323094 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.323202 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:04.329014 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:04.329028 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:04.329046 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:04.329094 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:04.333264 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:04.334192 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:04.334200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:04.334204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:04.334395 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:04.334874 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:04.334879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:04.334882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:04.335331 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:04.335617 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:04.336072 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 14:57:04.336563 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 14:57:04.337514 4960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:04.338456 4763 catalog_manager.cc:4841] TS e3351a801d0c48b4aede1408ba3893d3 (127.17.133.195:43117): tablet d8e9abda4fa842228b99a66d90d15205 (table test-table [id=bd06fc91ce5d49c9b143a6c3a542820c]) successfully deleted
I20260925 14:57:04.438457 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 4837
I20260925 14:57:04.456333 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 4971
Sep 25 14:57:04 dist-test-slave-v5vl krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.194@KRBTEST.COM
I20260925 14:57:04.470506 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:04.457212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.17.133.194:32797)
0925 14:57:04.457377 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:04.457382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:04.457398 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:57:04.457555 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:04.457558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:04.457617 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:04.457697 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:04.458878 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.459734 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.460978 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.461351 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.461443 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:04.465428 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:04.465443 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:04.465459 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:04.465496 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:04.468098 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:04.468796 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:04.468803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:04.468805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:04.468938 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:04.469247 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:04.469250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:04.469252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:04.469589 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:04.469806 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:04.470004 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 14:57:04.470157 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:57:04.471225 5094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:04.562109 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 4971
I20260925 14:57:04.578984 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 5104
Sep 25 14:57:04 dist-test-slave-v5vl krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.195@KRBTEST.COM
I20260925 14:57:04.594539 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:04.579809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.17.133.195:43117)
0925 14:57:04.580001 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:04.580006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:04.580025 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:57:04.580209 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:04.580213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:04.580274 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:04.580388 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:04.581744 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.582450 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.584012 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:04.584416 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:04.584559 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:04.588903 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:04.588919 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:04.588934 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:04.588976 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:04.591601 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:04.592281 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:04.592289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:04.592291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:04.592448 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:04.592962 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:04.592967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:04.592972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:04.593352 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:04.593653 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:04.593979 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 14:57:04.594196 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:57:04.595165 5227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:04.686695 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 5104
I20260925 14:57:04.703168 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 4744
I20260925 14:57:04.704232 4806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:04.813283 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 4744
2026-09-25T14:57:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4511 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T14:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:04Z Disabled control of system clock
I20260925 14:57:04.888044 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:33901
--webserver_interface=127.17.133.254
--webserver_port=0
--builtin_ntp_servers=127.17.133.212:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:57:05.209161 5286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:05.209424 5286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:05.209512 5286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:05.216225 5286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:05.216320 5286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:05.216355 5286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:05.216404 5286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:05.216446 5286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:05.225358 5286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:33901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:05.226540 5286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:05.228341 5286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:05.239362 5294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:05.239380 5291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:05.239377 5292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:05.240576 5286 server_base.cc:1034] running on GCE node
I20260925 14:57:05.241255 5286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:05.242535 5286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:05.243738 5286 hybrid_clock.cc:648] HybridClock initialized: now 1790348225243665 us; error 47 us; skew 500 ppm
I20260925 14:57:05.244024 5286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:57:05.246083 5286 webserver.cc:460] Webserver started at http://127.17.133.254:43397/ using document root <none> and password file <none>
I20260925 14:57:05.246680 5286 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:05.246757 5286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:05.246999 5286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:05.249665 5286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "8af2053bebf5420a9c3be40d2c0be30e"
format_stamp: "Formatted at 2026-09-25 14:57:05 on dist-test-slave-v5vl"
server_key: "5fc47d33de7221fb1f2de07f499ab14f"
server_key_iv: "86c809e48915d0a8ea67753c4e3c8548"
server_key_version: "encryptionkey@0"
I20260925 14:57:05.250329 5286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "8af2053bebf5420a9c3be40d2c0be30e"
format_stamp: "Formatted at 2026-09-25 14:57:05 on dist-test-slave-v5vl"
server_key: "5fc47d33de7221fb1f2de07f499ab14f"
server_key_iv: "86c809e48915d0a8ea67753c4e3c8548"
server_key_version: "encryptionkey@0"
I20260925 14:57:05.254988 5286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:57:05.258169 5300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:05.259474 5286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:57:05.259603 5286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "8af2053bebf5420a9c3be40d2c0be30e"
format_stamp: "Formatted at 2026-09-25 14:57:05 on dist-test-slave-v5vl"
server_key: "5fc47d33de7221fb1f2de07f499ab14f"
server_key_iv: "86c809e48915d0a8ea67753c4e3c8548"
server_key_version: "encryptionkey@0"
I20260925 14:57:05.259764 5286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:05.273090 5286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:05.273772 5286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:05.274003 5286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:05.291440 5286 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:33901
I20260925 14:57:05.291457 5351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:33901 every 8 connection(s)
I20260925 14:57:05.292822 5286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:05.296245 5352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:05.298714 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 5286
I20260925 14:57:05.298892 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:05.299464 17943 external_mini_cluster.cc:1428] Setting key 75ee5719f4580bd13507ca5563b09b65
I20260925 14:57:05.307677 5352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e: Bootstrap starting.
I20260925 14:57:05.310880 5352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:05.311986 5352 log.cc:824] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:05.312263 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:05.301331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.17.133.254:33901)
0925 14:57:05.302309 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:05.302320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:05.302358 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:57:05.303917 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:05.303938 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:05.304359 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:05.304740 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:05.305291 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:05.306517 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:05.307862 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:05.308533 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:05.308788 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:05.309971 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:05.309978 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:05.309991 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:05.310505 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:05.310594 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:05.310615 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:05.310741 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:05.310985 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 14:57:05.311325 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":396,"threads_started":1}
I20260925 14:57:05.314930 5352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e: No bootstrap required, opened a new log
I20260925 14:57:05.322336 5352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2053bebf5420a9c3be40d2c0be30e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33901 } }
I20260925 14:57:05.322682 5352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:05.322795 5352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af2053bebf5420a9c3be40d2c0be30e, State: Initialized, Role: FOLLOWER
I20260925 14:57:05.323383 5352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2053bebf5420a9c3be40d2c0be30e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33901 } }
I20260925 14:57:05.323619 5352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:05.323750 5352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:05.323916 5352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:05.326295 5352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2053bebf5420a9c3be40d2c0be30e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33901 } }
I20260925 14:57:05.326767 5352 leader_election.cc:302] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8af2053bebf5420a9c3be40d2c0be30e; no voters:
I20260925 14:57:05.327306 5352 leader_election.cc:288] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:05.327462 5357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:05.328146 5357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [term 1 LEADER]: Becoming Leader. State: Replica: 8af2053bebf5420a9c3be40d2c0be30e, State: Running, Role: LEADER
I20260925 14:57:05.328725 5357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2053bebf5420a9c3be40d2c0be30e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33901 } }
I20260925 14:57:05.329370 5352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:05.331506 5358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8af2053bebf5420a9c3be40d2c0be30e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2053bebf5420a9c3be40d2c0be30e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33901 } } }
I20260925 14:57:05.331883 5358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:05.332856 5359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8af2053bebf5420a9c3be40d2c0be30e. Latest consensus state: current_term: 1 leader_uuid: "8af2053bebf5420a9c3be40d2c0be30e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2053bebf5420a9c3be40d2c0be30e" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 33901 } } }
I20260925 14:57:05.333237 5359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:05.333881 5364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:05.339293 5364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:05.347496 5364 catalog_manager.cc:1347] Generated new cluster ID: bfecdd136d8d4e939b6a6524f37a2722
I20260925 14:57:05.347604 5364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:05.362768 5364 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:05.364519 5364 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:05.374428 5364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e: Generated new TSK 0
I20260925 14:57:05.375129 5364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:05.389119 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.193:0
--local_ip_for_outbound_sockets=127.17.133.193
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33901
--builtin_ntp_servers=127.17.133.212:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:57:05.718153 5376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:05.718436 5376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:05.718525 5376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:05.725456 5376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:05.725549 5376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:05.725651 5376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.193
I20260925 14:57:05.734227 5376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.133.193
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.193
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:05.735412 5376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:05.737128 5376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:05.750268 5381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:05.750312 5384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:05.750360 5382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:05.753914 5376 server_base.cc:1034] running on GCE node
I20260925 14:57:05.754668 5376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:05.755754 5376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:05.756983 5376 hybrid_clock.cc:648] HybridClock initialized: now 1790348225756920 us; error 59 us; skew 500 ppm
I20260925 14:57:05.757311 5376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:57:05.759640 5376 webserver.cc:460] Webserver started at http://127.17.133.193:38985/ using document root <none> and password file <none>
I20260925 14:57:05.760361 5376 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:05.760499 5376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:05.760860 5376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:05.764006 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/data/instance:
uuid: "a82b8e57a10c46efb04ef6e78aee2208"
format_stamp: "Formatted at 2026-09-25 14:57:05 on dist-test-slave-v5vl"
server_key: "cd37c1fa4f531720a5049fa38a0f78b3"
server_key_iv: "8c3d002e28a51b0f504dff4fe77c80a4"
server_key_version: "encryptionkey@0"
I20260925 14:57:05.764894 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance:
uuid: "a82b8e57a10c46efb04ef6e78aee2208"
format_stamp: "Formatted at 2026-09-25 14:57:05 on dist-test-slave-v5vl"
server_key: "cd37c1fa4f531720a5049fa38a0f78b3"
server_key_iv: "8c3d002e28a51b0f504dff4fe77c80a4"
server_key_version: "encryptionkey@0"
I20260925 14:57:05.769836 5376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 14:57:05.773188 5390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:05.774425 5376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:57:05.774598 5376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/wal
uuid: "a82b8e57a10c46efb04ef6e78aee2208"
format_stamp: "Formatted at 2026-09-25 14:57:05 on dist-test-slave-v5vl"
server_key: "cd37c1fa4f531720a5049fa38a0f78b3"
server_key_iv: "8c3d002e28a51b0f504dff4fe77c80a4"
server_key_version: "encryptionkey@0"
I20260925 14:57:05.774803 5376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:05.796267 5376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:05.797138 5376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:05.797448 5376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:05.798408 5376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:05.799882 5376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:05.800019 5376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:05.800143 5376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:05.800228 5376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:05.833128 5376 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.193:43539
I20260925 14:57:05.833166 5502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.193:43539 every 8 connection(s)
I20260925 14:57:05.834623 5376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:05.842555 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 5376
I20260925 14:57:05.842742 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:05.843173 17943 external_mini_cluster.cc:1428] Setting key e71debd065793d0a8f2eb589a0255299
I20260925 14:57:05.845609 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.194:0
--local_ip_for_outbound_sockets=127.17.133.194
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33901
--builtin_ntp_servers=127.17.133.212:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:57:05.847136 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:05.837216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.193:46223 (local address 127.17.133.254:33901)
0925 14:57:05.837623 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:05.837628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:05.838787 (+ 1159us) server_negotiation.cc:407] Connection header received
0925 14:57:05.840551 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:05.840559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:05.840679 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:05.840798 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:05.841835 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:05.842550 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:05.844274 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:05.844913 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:05.845053 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:05.846335 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:05.846341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:05.846344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:05.846360 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:05.846398 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:05.846410 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:05.846512 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:05.846716 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 14:57:05.846804 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":213,"threads_started":1}
I20260925 14:57:05.848141 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:05.837802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33901 (local address 127.17.133.193:46223)
0925 14:57:05.838644 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:05.838678 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:05.840290 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:05.841034 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:05.841042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:05.841125 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:05.841679 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:05.841694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:05.842690 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:05.842695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:05.844134 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:05.844145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:05.845129 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:05.845139 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:05.845373 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:05.846044 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:05.846046 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:05.846082 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:05.846155 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:05.846510 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:05.846522 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:05.846532 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:57:05.846706 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 14:57:05.847076 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":249,"threads_started":1}
I20260925 14:57:05.849704 5503 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33901
I20260925 14:57:05.849996 5503 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:05.850684 5503 heartbeater.cc:511] Master 127.17.133.254:33901 requested a full tablet report, sending...
I20260925 14:57:05.852864 5317 ts_manager.cc:194] Registered new tserver with Master: a82b8e57a10c46efb04ef6e78aee2208 (127.17.133.193:43539)
I20260925 14:57:05.854413 5317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.193:46223
W20260925 14:57:06.168890 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:06.169193 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:06.169353 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:06.180487 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:06.180632 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:06.180819 5507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.194
I20260925 14:57:06.190083 5507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.133.194
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.194
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:06.191394 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:06.193219 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:06.205963 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:06.206029 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:06.205933 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:06.206745 5507 server_base.cc:1034] running on GCE node
I20260925 14:57:06.207319 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:06.208276 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:06.209496 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790348226209452 us; error 62 us; skew 500 ppm
I20260925 14:57:06.209825 5507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:57:06.212019 5507 webserver.cc:460] Webserver started at http://127.17.133.194:46391/ using document root <none> and password file <none>
I20260925 14:57:06.212785 5507 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:06.212916 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:06.213218 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:06.215947 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/data/instance:
uuid: "786c3a33e69048e593d300e76b9b60a3"
format_stamp: "Formatted at 2026-09-25 14:57:06 on dist-test-slave-v5vl"
server_key: "ac0a2f40260d657bf88f10e36281d5f0"
server_key_iv: "e85ea49fc204667297cf51ec16bee974"
server_key_version: "encryptionkey@0"
I20260925 14:57:06.216770 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance:
uuid: "786c3a33e69048e593d300e76b9b60a3"
format_stamp: "Formatted at 2026-09-25 14:57:06 on dist-test-slave-v5vl"
server_key: "ac0a2f40260d657bf88f10e36281d5f0"
server_key_iv: "e85ea49fc204667297cf51ec16bee974"
server_key_version: "encryptionkey@0"
I20260925 14:57:06.221691 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:57:06.224973 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:06.226183 5507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:57:06.226353 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/wal
uuid: "786c3a33e69048e593d300e76b9b60a3"
format_stamp: "Formatted at 2026-09-25 14:57:06 on dist-test-slave-v5vl"
server_key: "ac0a2f40260d657bf88f10e36281d5f0"
server_key_iv: "e85ea49fc204667297cf51ec16bee974"
server_key_version: "encryptionkey@0"
I20260925 14:57:06.226570 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:06.250259 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:06.251099 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:06.251412 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:06.252359 5507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:06.253834 5507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:06.253959 5507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:06.254091 5507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:06.254180 5507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:06.286931 5507 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.194:38111
I20260925 14:57:06.287020 5633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.194:38111 every 8 connection(s)
I20260925 14:57:06.288205 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:06.291097 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 5507
I20260925 14:57:06.291275 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:06.291846 17943 external_mini_cluster.cc:1428] Setting key 8620056a0c274f51d2a53ac948abffda
I20260925 14:57:06.296509 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.133.195:0
--local_ip_for_outbound_sockets=127.17.133.195
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33901
--builtin_ntp_servers=127.17.133.212:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:57:06.300976 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.290752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.194:59021 (local address 127.17.133.254:33901)
0925 14:57:06.290959 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:06.290966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:06.292671 (+ 1705us) server_negotiation.cc:407] Connection header received
0925 14:57:06.294447 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:06.294453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:06.294537 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:06.294641 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:06.295922 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.296766 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.298248 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.298662 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.298793 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.300120 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:06.300125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:06.300129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:06.300146 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:06.300177 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:06.300188 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:06.300284 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:06.300533 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 14:57:06.300653 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 14:57:06.302026 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.291433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33901 (local address 127.17.133.194:59021)
0925 14:57:06.292492 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:06.292536 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:06.294195 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:06.294897 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:06.294909 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:06.295039 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:06.295759 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.295778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.296902 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.296913 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.298108 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.298119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.298889 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.298896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.299117 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.299808 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:06.299810 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:06.299850 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:06.299925 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:06.300282 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:06.300295 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:06.300305 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:57:06.300511 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 14:57:06.300911 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":317,"threads_started":1}
I20260925 14:57:06.303388 5634 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33901
I20260925 14:57:06.303694 5634 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:06.304389 5634 heartbeater.cc:511] Master 127.17.133.254:33901 requested a full tablet report, sending...
I20260925 14:57:06.305897 5317 ts_manager.cc:194] Registered new tserver with Master: 786c3a33e69048e593d300e76b9b60a3 (127.17.133.194:38111)
I20260925 14:57:06.307057 5317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.194:59021
W20260925 14:57:06.624523 5637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:06.624850 5637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:06.625016 5637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:06.632366 5637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:06.632459 5637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:06.632686 5637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.133.195
I20260925 14:57:06.642017 5637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.133.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.133.195
--webserver_port=0
--tserver_master_addrs=127.17.133.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.133.195
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:06.643304 5637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:06.645131 5637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:06.658082 5642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:06.658104 5643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:06.658128 5645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:06.661976 5637 server_base.cc:1034] running on GCE node
I20260925 14:57:06.662563 5637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:06.663592 5637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:06.664812 5637 hybrid_clock.cc:648] HybridClock initialized: now 1790348226664749 us; error 57 us; skew 500 ppm
I20260925 14:57:06.665132 5637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:57:06.667263 5637 webserver.cc:460] Webserver started at http://127.17.133.195:36971/ using document root <none> and password file <none>
I20260925 14:57:06.667935 5637 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:06.668063 5637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:06.668371 5637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:06.671221 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/data/instance:
uuid: "958e10447a0b4ef3939f017071481c39"
format_stamp: "Formatted at 2026-09-25 14:57:06 on dist-test-slave-v5vl"
server_key: "00cd52cb4b6a16733ac056d14fc47bb7"
server_key_iv: "4528048c30e29b2645c5e816965ad326"
server_key_version: "encryptionkey@0"
I20260925 14:57:06.671996 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance:
uuid: "958e10447a0b4ef3939f017071481c39"
format_stamp: "Formatted at 2026-09-25 14:57:06 on dist-test-slave-v5vl"
server_key: "00cd52cb4b6a16733ac056d14fc47bb7"
server_key_iv: "4528048c30e29b2645c5e816965ad326"
server_key_version: "encryptionkey@0"
I20260925 14:57:06.676997 5637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 14:57:06.680619 5651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:06.681838 5637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:57:06.682027 5637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/wal
uuid: "958e10447a0b4ef3939f017071481c39"
format_stamp: "Formatted at 2026-09-25 14:57:06 on dist-test-slave-v5vl"
server_key: "00cd52cb4b6a16733ac056d14fc47bb7"
server_key_iv: "4528048c30e29b2645c5e816965ad326"
server_key_version: "encryptionkey@0"
I20260925 14:57:06.682262 5637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:06.702803 5637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:06.703562 5637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:06.703814 5637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:06.704759 5637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:06.706148 5637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:06.706233 5637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:06.706332 5637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:06.706388 5637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:06.738420 5637 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.195:44431
I20260925 14:57:06.738507 5763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.195:44431 every 8 connection(s)
I20260925 14:57:06.739727 5637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:06.744747 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 5637
I20260925 14:57:06.744935 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:06.745455 17943 external_mini_cluster.cc:1428] Setting key 2ae778e161403c5910ea7cfb65ee519d
I20260925 14:57:06.752903 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.742366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:35903 (local address 127.17.133.254:33901)
0925 14:57:06.742585 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:06.742592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:06.743964 (+ 1372us) server_negotiation.cc:407] Connection header received
0925 14:57:06.746354 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:06.746361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:06.746462 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:06.746581 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:06.747733 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.748679 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.750219 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.750643 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.750779 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.752063 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:06.752068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:06.752071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:06.752090 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:06.752125 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:06.752137 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:06.752232 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:06.752448 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:57:06.752556 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 14:57:06.753919 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.742919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:33901 (local address 127.17.133.195:35903)
0925 14:57:06.743797 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:06.743836 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:06.746077 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:06.746823 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:06.746835 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:06.746959 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:06.747569 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.747586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.748814 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.748820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.750071 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.750083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.750861 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.750868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.751106 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.751739 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:06.751742 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:06.751783 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:06.751863 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:06.752230 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:06.752242 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:06.752252 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:57:06.752439 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 14:57:06.752870 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260925 14:57:06.755225 5764 heartbeater.cc:347] Connected to a master server at 127.17.133.254:33901
I20260925 14:57:06.755545 5764 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:06.756294 5764 heartbeater.cc:511] Master 127.17.133.254:33901 requested a full tablet report, sending...
I20260925 14:57:06.757786 5317 ts_manager.cc:194] Registered new tserver with Master: 958e10447a0b4ef3939f017071481c39 (127.17.133.195:44431)
I20260925 14:57:06.758760 5317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.133.195:35903
I20260925 14:57:06.760756 17943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:06.772017 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.764971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.17.133.254:33901)
0925 14:57:06.765208 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:06.765216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:06.766575 (+ 1359us) server_negotiation.cc:407] Connection header received
0925 14:57:06.766788 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:06.766794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:06.766877 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:06.766988 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:06.767507 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.768219 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.769635 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.770061 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.770194 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.771270 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:06.771275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:06.771278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:06.771297 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:06.771339 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:06.771351 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:06.771453 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:06.771601 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:57:06.771707 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:57:06.775913 5316 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:06.782932 5316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:06.804970 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.794918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.195:44431 (local address 127.0.0.1:51294)
0925 14:57:06.795735 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:06.795757 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:06.795915 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:06.798388 (+ 2473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:06.798393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:06.798416 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:06.798755 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.798764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.800434 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.800442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.802225 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.802237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.804063 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.804069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.804231 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.804268 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:57:06.804349 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:57:06.804494 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":19,"thread_start_us":864,"threads_started":1}
I20260925 14:57:06.805588 5777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.794589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.17.133.193:43539)
0925 14:57:06.795306 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:06.795315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:06.795800 (+ 485us) server_negotiation.cc:407] Connection header received
0925 14:57:06.796192 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:06.796204 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:06.796487 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:06.796691 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:06.797730 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.799367 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.802350 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.803861 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.804037 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.804278 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:06.805067 (+ 789us) server_negotiation.cc:299] Negotiation successful
0925 14:57:06.805240 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":335,"threads_started":1}
I20260925 14:57:06.805639 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.795521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.17.133.195:44431)
0925 14:57:06.797880 (+ 2359us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:06.797888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:06.797918 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:57:06.798009 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:06.798018 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:06.798204 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:06.798386 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:06.798913 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.800295 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.802466 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.803843 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.804816 (+ 973us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.805035 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:06.805111 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:57:06.805262 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2176,"thread_start_us":2019,"threads_started":1}
I20260925 14:57:06.805043 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.796045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:38111 (local address 127.0.0.1:34044)
0925 14:57:06.796681 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:06.796701 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:06.796805 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:06.797876 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:06.797881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:06.797905 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:06.798261 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.798273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.800101 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.800108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.802345 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.802361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.804182 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.804188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.804382 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.804414 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 14:57:06.804482 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:57:06.804627 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":1,"thread_start_us":391,"threads_started":1}
I20260925 14:57:06.805948 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.796502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.17.133.194:38111)
0925 14:57:06.797277 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:06.797303 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:06.797329 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:57:06.797426 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:06.797436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:06.797675 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:06.797861 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:06.798478 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.799937 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.802551 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.803961 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.805191 (+ 1230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.805375 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:06.805436 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:57:06.805597 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":473,"threads_started":1}
I20260925 14:57:06.806080 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.794429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43539 (local address 127.0.0.1:48970)
0925 14:57:06.795651 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:06.795693 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:06.795918 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:06.797057 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:06.797064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:06.797115 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:06.797530 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.797548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.799539 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.799544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.802114 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.802131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.804397 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.804403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.804613 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.804640 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 14:57:06.804717 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:57:06.804829 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":1,"thread_start_us":791,"threads_started":1}
I20260925 14:57:06.809399 5569 tablet_service.cc:1467] Processing CreateTablet for tablet d8a608e6da3c4effaca693642ecd8c0e (DEFAULT_TABLE table=test-table [id=3bd2cb6f217040509cd93fc083300b7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:06.809499 5699 tablet_service.cc:1467] Processing CreateTablet for tablet d8a608e6da3c4effaca693642ecd8c0e (DEFAULT_TABLE table=test-table [id=3bd2cb6f217040509cd93fc083300b7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:06.809507 5438 tablet_service.cc:1467] Processing CreateTablet for tablet d8a608e6da3c4effaca693642ecd8c0e (DEFAULT_TABLE table=test-table [id=3bd2cb6f217040509cd93fc083300b7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:06.810609 5569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a608e6da3c4effaca693642ecd8c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:06.810799 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a608e6da3c4effaca693642ecd8c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:06.810801 5438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a608e6da3c4effaca693642ecd8c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:06.821602 5782 tablet_bootstrap.cc:492] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: Bootstrap starting.
I20260925 14:57:06.824882 5782 tablet_bootstrap.cc:654] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:06.825692 5784 tablet_bootstrap.cc:492] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: Bootstrap starting.
I20260925 14:57:06.825769 5783 tablet_bootstrap.cc:492] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208: Bootstrap starting.
I20260925 14:57:06.827365 5782 log.cc:824] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:06.828538 5783 tablet_bootstrap.cc:654] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:06.829345 5784 tablet_bootstrap.cc:654] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:06.829659 5783 log.cc:824] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:06.830773 5784 log.cc:824] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:06.837357 5784 tablet_bootstrap.cc:492] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: No bootstrap required, opened a new log
I20260925 14:57:06.837725 5784 ts_tablet_manager.cc:1397] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260925 14:57:06.838291 5782 tablet_bootstrap.cc:492] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: No bootstrap required, opened a new log
I20260925 14:57:06.838645 5782 ts_tablet_manager.cc:1397] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20260925 14:57:06.848416 5784 raft_consensus.cc:348] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } }
I20260925 14:57:06.850103 5784 raft_consensus.cc:374] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:06.850252 5784 raft_consensus.cc:729] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958e10447a0b4ef3939f017071481c39, State: Initialized, Role: FOLLOWER
I20260925 14:57:06.850242 5782 raft_consensus.cc:348] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } }
I20260925 14:57:06.850677 5782 raft_consensus.cc:374] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:06.850800 5782 raft_consensus.cc:729] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786c3a33e69048e593d300e76b9b60a3, State: Initialized, Role: FOLLOWER
I20260925 14:57:06.851050 5783 tablet_bootstrap.cc:492] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208: No bootstrap required, opened a new log
I20260925 14:57:06.851164 5784 consensus_queue.cc:260] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } }
I20260925 14:57:06.851356 5783 ts_tablet_manager.cc:1397] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.018s
I20260925 14:57:06.851462 5782 consensus_queue.cc:260] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } }
I20260925 14:57:06.853287 5764 heartbeater.cc:503] Master 127.17.133.254:33901 was elected leader, sending a full tablet report...
I20260925 14:57:06.855825 5784 ts_tablet_manager.cc:1428] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: Time spent starting tablet: real 0.018s user 0.012s sys 0.001s
I20260925 14:57:06.856842 5634 heartbeater.cc:503] Master 127.17.133.254:33901 was elected leader, sending a full tablet report...
I20260925 14:57:06.857439 5782 ts_tablet_manager.cc:1428] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
I20260925 14:57:06.857887 5503 heartbeater.cc:503] Master 127.17.133.254:33901 was elected leader, sending a full tablet report...
I20260925 14:57:06.860525 5783 raft_consensus.cc:348] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } }
I20260925 14:57:06.861032 5783 raft_consensus.cc:374] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:06.861176 5783 raft_consensus.cc:729] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82b8e57a10c46efb04ef6e78aee2208, State: Initialized, Role: FOLLOWER
I20260925 14:57:06.861820 5783 consensus_queue.cc:260] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } }
I20260925 14:57:06.863276 5783 ts_tablet_manager.cc:1428] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260925 14:57:06.961613 5789 raft_consensus.cc:482] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:06.961876 5789 raft_consensus.cc:504] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } }
I20260925 14:57:06.963732 5789 leader_election.cc:288] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a82b8e57a10c46efb04ef6e78aee2208 (127.17.133.193:43539), 786c3a33e69048e593d300e76b9b60a3 (127.17.133.194:38111)
I20260925 14:57:06.970207 5777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.964176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:51609 (local address 127.17.133.193:43539)
0925 14:57:06.964414 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:06.964421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:06.964442 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:57:06.964533 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:06.964540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:06.964695 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:06.964797 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:06.965403 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.966466 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.968296 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.969423 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.969538 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.969585 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:06.969946 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 14:57:06.970032 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 14:57:06.970358 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.964008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43539 (local address 127.17.133.195:51609)
0925 14:57:06.964227 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:06.964247 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:06.964361 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:06.964996 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:06.965004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:06.965034 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:06.965218 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.965226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.966595 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.966600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.968145 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.968156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.969565 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.969571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.969767 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.969807 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 14:57:06.969899 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 14:57:06.970031 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 14:57:06.971037 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.964412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.194:38111 (local address 127.17.133.195:58871)
0925 14:57:06.965202 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:06.965219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:06.965345 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:06.965635 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:06.965639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:06.965664 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:06.966073 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.966085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.967308 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.967312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.968976 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:06.968990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.970373 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.970382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:06.970564 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.970592 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 14:57:06.970720 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 14:57:06.970798 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":486,"threads_started":1}
I20260925 14:57:06.971151 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:06.964722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.133.195:58871 (local address 127.17.133.194:38111)
0925 14:57:06.964987 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:06.964993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:06.965313 (+ 320us) server_negotiation.cc:407] Connection header received
0925 14:57:06.965470 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:06.965473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:06.965527 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:06.965593 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:06.966225 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.967173 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.969153 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:06.970191 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:06.970337 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:06.970413 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:06.970774 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 14:57:06.970895 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260925 14:57:06.971488 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a608e6da3c4effaca693642ecd8c0e" candidate_uuid: "958e10447a0b4ef3939f017071481c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82b8e57a10c46efb04ef6e78aee2208" is_pre_election: true
I20260925 14:57:06.972100 5458 raft_consensus.cc:2454] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 958e10447a0b4ef3939f017071481c39 in term 0.
I20260925 14:57:06.972075 5589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a608e6da3c4effaca693642ecd8c0e" candidate_uuid: "958e10447a0b4ef3939f017071481c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786c3a33e69048e593d300e76b9b60a3" is_pre_election: true
I20260925 14:57:06.972627 5589 raft_consensus.cc:2454] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 958e10447a0b4ef3939f017071481c39 in term 0.
I20260925 14:57:06.972810 5655 leader_election.cc:302] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 958e10447a0b4ef3939f017071481c39, a82b8e57a10c46efb04ef6e78aee2208; no voters:
I20260925 14:57:06.973204 5789 raft_consensus.cc:2781] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:06.973342 5789 raft_consensus.cc:482] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:06.973469 5789 raft_consensus.cc:3037] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:06.975657 5789 raft_consensus.cc:504] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } }
I20260925 14:57:06.976212 5789 leader_election.cc:288] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [CANDIDATE]: Term 1 election: Requested vote from peers a82b8e57a10c46efb04ef6e78aee2208 (127.17.133.193:43539), 786c3a33e69048e593d300e76b9b60a3 (127.17.133.194:38111)
I20260925 14:57:06.976728 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a608e6da3c4effaca693642ecd8c0e" candidate_uuid: "958e10447a0b4ef3939f017071481c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82b8e57a10c46efb04ef6e78aee2208"
I20260925 14:57:06.976776 5589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a608e6da3c4effaca693642ecd8c0e" candidate_uuid: "958e10447a0b4ef3939f017071481c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786c3a33e69048e593d300e76b9b60a3"
I20260925 14:57:06.976943 5458 raft_consensus.cc:3037] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:06.976961 5589 raft_consensus.cc:3037] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:06.979391 5589 raft_consensus.cc:2454] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 958e10447a0b4ef3939f017071481c39 in term 1.
I20260925 14:57:06.979396 5458 raft_consensus.cc:2454] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 958e10447a0b4ef3939f017071481c39 in term 1.
I20260925 14:57:06.979806 5655 leader_election.cc:302] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786c3a33e69048e593d300e76b9b60a3, 958e10447a0b4ef3939f017071481c39; no voters:
I20260925 14:57:06.980163 5789 raft_consensus.cc:2781] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:06.980643 5789 raft_consensus.cc:686] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 1 LEADER]: Becoming Leader. State: Replica: 958e10447a0b4ef3939f017071481c39, State: Running, Role: LEADER
I20260925 14:57:06.981204 5789 consensus_queue.cc:237] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } }
I20260925 14:57:06.985736 5315 catalog_manager.cc:5495] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 958e10447a0b4ef3939f017071481c39 (127.17.133.195). New cstate: current_term: 1 leader_uuid: "958e10447a0b4ef3939f017071481c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "958e10447a0b4ef3939f017071481c39" member_type: VOTER last_known_addr { host: "127.17.133.195" port: 44431 } health_report { overall_health: HEALTHY } } }
W20260925 14:57:06.991317 5765 tablet.cc:2369] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:07.040305 5635 tablet.cc:2369] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:07.045961 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.040603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.17.133.195:44431)
0925 14:57:07.040787 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:07.040794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:07.040876 (+ 82us) server_negotiation.cc:407] Connection header received
0925 14:57:07.041126 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:07.041131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:07.041218 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:07.041371 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:07.041740 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:07.042450 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:07.043825 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:07.044245 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:07.044372 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:07.044902 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:07.045398 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:07.045560 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:07.045636 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:57:07.045712 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:57:07.053488 5589 raft_consensus.cc:1261] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [term 1 FOLLOWER]: Refusing update from remote peer 958e10447a0b4ef3939f017071481c39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:07.053541 5458 raft_consensus.cc:1261] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208 [term 1 FOLLOWER]: Refusing update from remote peer 958e10447a0b4ef3939f017071481c39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:07.054369 5792 consensus_queue.cc:1035] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786c3a33e69048e593d300e76b9b60a3" member_type: VOTER last_known_addr { host: "127.17.133.194" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:07.054726 5789 consensus_queue.cc:1035] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82b8e57a10c46efb04ef6e78aee2208" member_type: VOTER last_known_addr { host: "127.17.133.193" port: 43539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:07.066113 5795 mvcc.cc:205] Tried to move back new op lower bound from 7333266338001920000 to 7333266337718964224. Current Snapshot: MvccSnapshot[applied={T|T < 7333266338001920000}]
I20260925 14:57:07.071691 5796 mvcc.cc:205] Tried to move back new op lower bound from 7333266338001920000 to 7333266337718964224. Current Snapshot: MvccSnapshot[applied={T|T < 7333266338001920000}]
I20260925 14:57:07.079809 5315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:07.080067 5315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:07.083513 5315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8af2053bebf5420a9c3be40d2c0be30e: Sending DeleteTablet for 3 replicas of tablet d8a608e6da3c4effaca693642ecd8c0e
I20260925 14:57:07.084743 5569 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a608e6da3c4effaca693642ecd8c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:07 UTC) from {username='slave'} at 127.0.0.1:34044
I20260925 14:57:07.084949 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a608e6da3c4effaca693642ecd8c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:07 UTC) from {username='slave'} at 127.0.0.1:48970
I20260925 14:57:07.085469 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 5376
I20260925 14:57:07.085420 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a608e6da3c4effaca693642ecd8c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:07 UTC) from {username='slave'} at 127.0.0.1:51294
I20260925 14:57:07.107697 5808 tablet_replica.cc:330] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: stopping tablet replica
I20260925 14:57:07.108155 5777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.086790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.17.133.193:43539)
0925 14:57:07.088149 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:07.088158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:07.098206 (+ 10048us) server_negotiation.cc:407] Connection header received
0925 14:57:07.098703 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:07.098711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:07.098836 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:07.098920 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:07.099675 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:07.101183 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:07.103498 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:07.104379 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:07.104492 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:07.106253 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:07.106269 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:07.106292 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:07.107236 (+ 944us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:07.107302 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:07.107328 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:07.107417 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:07.107681 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 14:57:07.107798 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198}
I20260925 14:57:07.108647 5808 raft_consensus.cc:2229] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:57:07.109350 5498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:07.109490 5808 raft_consensus.cc:2258] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:07.110723 5806 tablet_replica.cc:330] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: stopping tablet replica
I20260925 14:57:07.113166 5806 raft_consensus.cc:2229] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:07.113906 5806 raft_consensus.cc:2258] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:07.114557 5808 ts_tablet_manager.cc:1905] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:07.126981 5806 ts_tablet_manager.cc:1905] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:07.132999 5808 ts_tablet_manager.cc:1918] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:07.133267 5808 log.cc:1196] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-2/wal/wals/d8a608e6da3c4effaca693642ecd8c0e
I20260925 14:57:07.133882 5808 ts_tablet_manager.cc:1939] T d8a608e6da3c4effaca693642ecd8c0e P 958e10447a0b4ef3939f017071481c39: Deleting consensus metadata
I20260925 14:57:07.135872 5303 catalog_manager.cc:4841] TS 958e10447a0b4ef3939f017071481c39 (127.17.133.195:44431): tablet d8a608e6da3c4effaca693642ecd8c0e (table test-table [id=3bd2cb6f217040509cd93fc083300b7d]) successfully deleted
I20260925 14:57:07.136417 5806 ts_tablet_manager.cc:1918] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:07.136632 5806 log.cc:1196] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: Deleting WAL directory at /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347918228911-17943-0/minicluster-data/ts-1/wal/wals/d8a608e6da3c4effaca693642ecd8c0e
I20260925 14:57:07.137176 5806 ts_tablet_manager.cc:1939] T d8a608e6da3c4effaca693642ecd8c0e P 786c3a33e69048e593d300e76b9b60a3: Deleting consensus metadata
I20260925 14:57:07.139149 5304 catalog_manager.cc:4841] TS 786c3a33e69048e593d300e76b9b60a3 (127.17.133.194:38111): tablet d8a608e6da3c4effaca693642ecd8c0e (table test-table [id=3bd2cb6f217040509cd93fc083300b7d]) successfully deleted
I20260925 14:57:07.218169 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 5376
I20260925 14:57:07.218176 5807 tablet_replica.cc:330] T d8a608e6da3c4effaca693642ecd8c0e P a82b8e57a10c46efb04ef6e78aee2208: stopping tablet replica
W20260925 14:57:07.236488 5304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.133.193:43539 (error 108)
I20260925 14:57:07.236729 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 5507
I20260925 14:57:07.240061 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.239570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43539 (local address 127.0.0.1:48988)
0925 14:57:07.239770 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.239891 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 14:57:07.240306 5304 catalog_manager.cc:4584] TS a82b8e57a10c46efb04ef6e78aee2208 (127.17.133.193:43539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8a608e6da3c4effaca693642ecd8c0e: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43539: connect: Connection refused (error 111)
I20260925 14:57:07.244182 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.237991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.17.133.194:38111)
0925 14:57:07.238168 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:07.238175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:07.238197 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:57:07.238266 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:07.238272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:07.238395 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:07.238516 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:07.239064 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:07.239984 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:07.241338 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:07.241742 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:07.241869 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:07.242932 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:07.242940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:07.242952 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:07.243482 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:07.243533 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:07.243551 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:07.243661 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:07.243809 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:57:07.243918 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:57:07.244827 5629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:07.294103 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.293623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43539 (local address 127.0.0.1:49004)
0925 14:57:07.293794 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.293889 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 14:57:07.295341 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.294990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43539 (local address 127.0.0.1:49008)
0925 14:57:07.295116 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.295187 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:57:07.335271 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 5507
I20260925 14:57:07.354455 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 5637
I20260925 14:57:07.356520 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.355951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43539 (local address 127.0.0.1:49012)
0925 14:57:07.356218 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.356345 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260925 14:57:07.358464 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.358037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43539 (local address 127.0.0.1:49016)
0925 14:57:07.358210 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.358296 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 14:57:07.362126 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.355208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.17.133.195:44431)
0925 14:57:07.355433 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:07.355451 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:07.355472 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:57:07.355608 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:07.355614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:07.355722 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:07.355858 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:07.356317 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:07.357253 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:07.358949 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:07.359414 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:07.359575 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:07.360792 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:07.360800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:07.360814 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:07.361415 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:07.361471 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:07.361489 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:07.361601 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:07.361709 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:57:07.361802 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 14:57:07.362735 5759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:07.439138 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.438668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43539 (local address 127.0.0.1:42912)
0925 14:57:07.438879 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.438967 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 14:57:07.440286 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.439937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.193:43539 (local address 127.0.0.1:42926)
0925 14:57:07.440120 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.440187 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.193:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 14:57:07.483373 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 5637
I20260925 14:57:07.502389 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 5286
I20260925 14:57:07.503453 5347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:07.613956 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 5286
2026-09-25T14:57:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2800 ms)
[----------] 6 tests from EncryptionPolicyTest (21406 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:07 dist-test-slave-v5vl krb5kdc[5817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:07 dist-test-slave-v5vl krb5kdc[5817](info): set up 2 sockets
Sep 25 14:57:07 dist-test-slave-v5vl krb5kdc[5817](info): commencing operation
krb5kdc: starting...
W20260925 14:57:08.721774 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:08 dist-test-slave-v5vl krb5kdc[5817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:08.964252 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:09.284489 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:09.284771 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:09.284960 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:09.292310 5830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:09.292423 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:09.292469 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:09.292613 5830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:09.292702 5830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:09.302413 5830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:09.303750 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:09.305119 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:09.316885 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:09.316926 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:09.316926 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:09.318190 5830 server_base.cc:1034] running on GCE node
I20260925 14:57:09.319013 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:09.320415 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:09.321635 5830 hybrid_clock.cc:648] HybridClock initialized: now 1790348229321581 us; error 49 us; skew 500 ppm
I20260925 14:57:09.321985 5830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:09 dist-test-slave-v5vl krb5kdc[5817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:09.328912 5830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:09.330329 5830 webserver.cc:460] Webserver started at http://127.0.0.1:37291/ using document root <none> and password file <none>
I20260925 14:57:09.330986 5830 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:09.331127 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:09.331419 5830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:09.334299 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "fdeb2be69c1b4c74841a558b333fc1fa"
format_stamp: "Formatted at 2026-09-25 14:57:09 on dist-test-slave-v5vl"
server_key: "2e1a216628c88d9a71b8df5fd2efa81e"
server_key_iv: "2bd7331fecd01c9cf79208fc60f82fcf"
server_key_version: "encryptionkey@0"
I20260925 14:57:09.335045 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "fdeb2be69c1b4c74841a558b333fc1fa"
format_stamp: "Formatted at 2026-09-25 14:57:09 on dist-test-slave-v5vl"
server_key: "2e1a216628c88d9a71b8df5fd2efa81e"
server_key_iv: "2bd7331fecd01c9cf79208fc60f82fcf"
server_key_version: "encryptionkey@0"
I20260925 14:57:09.339941 5830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 14:57:09.343147 5845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:09.344358 5830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:09.344682 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "fdeb2be69c1b4c74841a558b333fc1fa"
format_stamp: "Formatted at 2026-09-25 14:57:09 on dist-test-slave-v5vl"
server_key: "2e1a216628c88d9a71b8df5fd2efa81e"
server_key_iv: "2bd7331fecd01c9cf79208fc60f82fcf"
server_key_version: "encryptionkey@0"
I20260925 14:57:09.344871 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:09.356992 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:09.361012 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:09.361301 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:09.379442 5830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45049
I20260925 14:57:09.379498 5896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45049 every 8 connection(s)
I20260925 14:57:09.380872 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:09.384536 5897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:09.388736 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 5830
I20260925 14:57:09.388947 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:09.389575 17943 external_mini_cluster.cc:1428] Setting key 04300b4c02e2a7b05b92f575f8c58234
I20260925 14:57:09.395568 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa: Bootstrap starting.
I20260925 14:57:09.399258 5897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:09.400350 5897 log.cc:824] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:09.403560 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa: No bootstrap required, opened a new log
Sep 25 14:57:09 dist-test-slave-v5vl krb5kdc[5817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:57:09.413388 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.391317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60341 (local address 127.0.0.1:45049)
0925 14:57:09.392076 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:09.392087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:09.392125 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:57:09.394029 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:09.394049 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:09.394392 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:09.394677 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:09.396383 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.397817 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.399617 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.400240 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.400376 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:09.405179 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:09.405188 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:09.405206 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:09.405257 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:09.409186 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:09.409845 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:09.409869 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:09.409877 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:09.410065 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:09.410446 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:09.410450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:09.410454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:09.411100 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:09.411398 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:09.411723 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 14:57:09.412055 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":369,"threads_started":1}
I20260925 14:57:09.413898 5897 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdeb2be69c1b4c74841a558b333fc1fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } }
I20260925 14:57:09.414276 5897 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:09.414387 5897 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdeb2be69c1b4c74841a558b333fc1fa, State: Initialized, Role: FOLLOWER
I20260925 14:57:09.414885 5897 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdeb2be69c1b4c74841a558b333fc1fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } }
I20260925 14:57:09.415118 5897 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:09.415253 5897 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:09.415512 5897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:09.417232 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:09.418359 5897 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdeb2be69c1b4c74841a558b333fc1fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } }
I20260925 14:57:09.418865 5897 leader_election.cc:302] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdeb2be69c1b4c74841a558b333fc1fa; no voters:
I20260925 14:57:09.419421 5897 leader_election.cc:288] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:09.419486 5902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:09.420888 5902 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [term 1 LEADER]: Becoming Leader. State: Replica: fdeb2be69c1b4c74841a558b333fc1fa, State: Running, Role: LEADER
I20260925 14:57:09.421506 5902 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdeb2be69c1b4c74841a558b333fc1fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } }
I20260925 14:57:09.421862 5897 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:09.424373 5903 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdeb2be69c1b4c74841a558b333fc1fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdeb2be69c1b4c74841a558b333fc1fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } } }
I20260925 14:57:09.424746 5903 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:09.424103 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdeb2be69c1b4c74841a558b333fc1fa. Latest consensus state: current_term: 1 leader_uuid: "fdeb2be69c1b4c74841a558b333fc1fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdeb2be69c1b4c74841a558b333fc1fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } } }
I20260925 14:57:09.425416 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:09.433539 5912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:09.438352 5912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:09.451490 5912 catalog_manager.cc:1347] Generated new cluster ID: 40b9d88d66e34ba69d16fc78c21f48f5
I20260925 14:57:09.451690 5912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:09.456981 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.424150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44529 (local address 127.0.0.1:45049)
0925 14:57:09.429761 (+ 5611us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:09.429769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:09.434911 (+ 5142us) server_negotiation.cc:407] Connection header received
0925 14:57:09.435407 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:09.435416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:09.435527 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:09.435631 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:09.438107 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.439702 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.441606 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.442302 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.442443 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:09.445508 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:09.445513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:09.445517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:09.445551 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:09.451478 (+ 5927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:09.452820 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:09.452827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:09.452831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:09.453006 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:09.453704 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:09.453712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:09.453717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:09.454216 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:09.455977 (+ 1761us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:09.456214 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 14:57:09.456463 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5446}
I20260925 14:57:09.493413 5912 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:09.494716 5912 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:09.502000 5912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdeb2be69c1b4c74841a558b333fc1fa: Generated new TSK 0
I20260925 14:57:09.502717 5912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:09.508103 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 5830
I20260925 14:57:09.509933 5892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:09.602010 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 5830
W20260925 14:57:09.618945 5815 connection.cc:535] client connection to 127.0.0.1:45049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45049): Connection reset by peer (error 104)
2026-09-25T14:57:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:09 dist-test-slave-v5vl krb5kdc[5930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:09 dist-test-slave-v5vl krb5kdc[5930](info): set up 2 sockets
Sep 25 14:57:09 dist-test-slave-v5vl krb5kdc[5930](info): commencing operation
krb5kdc: starting...
W20260925 14:57:10.692943 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:10 dist-test-slave-v5vl krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:10.935871 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:11.261301 5943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:11.261601 5943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:11.261783 5943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:11.269073 5943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:11.269181 5943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:11.269364 5943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:11.269469 5943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:11.269553 5943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:11.279585 5943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:11.280866 5943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:11.282172 5943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:11.293407 5948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:11.293323 5951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:11.293368 5949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:11.294266 5943 server_base.cc:1034] running on GCE node
I20260925 14:57:11.295018 5943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:11.296276 5943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:11.297484 5943 hybrid_clock.cc:648] HybridClock initialized: now 1790348231297430 us; error 51 us; skew 500 ppm
I20260925 14:57:11.297834 5943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:11 dist-test-slave-v5vl krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:11.300863 5943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:11.302312 5943 webserver.cc:460] Webserver started at http://127.0.0.1:44539/ using document root <none> and password file <none>
I20260925 14:57:11.302933 5943 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:11.303068 5943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:11.303354 5943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:11.306222 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "9f0f00189f0846409013500454a11f6a"
format_stamp: "Formatted at 2026-09-25 14:57:11 on dist-test-slave-v5vl"
server_key: "60d25e84510b324c5cf20c5ffd49fc38"
server_key_iv: "c5a7c6fc18451ecd6b5d32806d54e46e"
server_key_version: "encryptionkey@0"
I20260925 14:57:11.306967 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "9f0f00189f0846409013500454a11f6a"
format_stamp: "Formatted at 2026-09-25 14:57:11 on dist-test-slave-v5vl"
server_key: "60d25e84510b324c5cf20c5ffd49fc38"
server_key_iv: "c5a7c6fc18451ecd6b5d32806d54e46e"
server_key_version: "encryptionkey@0"
I20260925 14:57:11.311985 5943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:57:11.315358 5958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:11.316725 5943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:11.316900 5943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "9f0f00189f0846409013500454a11f6a"
format_stamp: "Formatted at 2026-09-25 14:57:11 on dist-test-slave-v5vl"
server_key: "60d25e84510b324c5cf20c5ffd49fc38"
server_key_iv: "c5a7c6fc18451ecd6b5d32806d54e46e"
server_key_version: "encryptionkey@0"
I20260925 14:57:11.317087 5943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:11.330204 5943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:11.334041 5943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:11.334314 5943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:11.351672 5943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41369
I20260925 14:57:11.351727 6009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41369 every 8 connection(s)
I20260925 14:57:11.352952 5943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:11.356282 6010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:11.359683 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 5943
I20260925 14:57:11.359853 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:11.360374 17943 external_mini_cluster.cc:1428] Setting key 4af874ae7b21186676d82675d763d612
I20260925 14:57:11.368331 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a: Bootstrap starting.
I20260925 14:57:11.371809 6010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:11.372979 6010 log.cc:824] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:11.373358 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:11.362096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53837 (local address 127.0.0.1:41369)
0925 14:57:11.362858 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:11.362870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:11.362909 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:57:11.364534 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:11.364555 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:11.365022 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:11.365341 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:11.365897 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:11.367150 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:11.368962 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:11.369585 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:11.369722 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:11.371034 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:11.371041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:11.371053 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:11.371555 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:11.371631 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:11.371654 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:11.371781 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:11.372142 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 14:57:11.372330 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":377,"threads_started":1}
I20260925 14:57:11.375774 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a: No bootstrap required, opened a new log
I20260925 14:57:11.377224 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:11.386246 6010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0f00189f0846409013500454a11f6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260925 14:57:11.386693 6010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:11.386801 6010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0f00189f0846409013500454a11f6a, State: Initialized, Role: FOLLOWER
I20260925 14:57:11.387405 6010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0f00189f0846409013500454a11f6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260925 14:57:11.387696 6010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:11.387833 6010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:11.388027 6010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:11.389544 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:11.382297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35805 (local address 127.0.0.1:41369)
0925 14:57:11.382493 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:11.382499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:11.382991 (+ 492us) server_negotiation.cc:407] Connection header received
0925 14:57:11.383262 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:11.383268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:11.383368 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:11.383482 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:11.384132 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:11.384985 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:11.386856 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:11.387430 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:11.387522 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:11.388767 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:11.388772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:11.388775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:11.388794 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:11.388834 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:11.388849 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:11.388934 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:11.389137 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 14:57:11.389177 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:57:11.390813 6010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0f00189f0846409013500454a11f6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260925 14:57:11.391292 6010 leader_election.cc:302] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f0f00189f0846409013500454a11f6a; no voters:
I20260925 14:57:11.391857 6010 leader_election.cc:288] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:11.391988 6020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:11.392699 6020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [term 1 LEADER]: Becoming Leader. State: Replica: 9f0f00189f0846409013500454a11f6a, State: Running, Role: LEADER
I20260925 14:57:11.393265 6020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0f00189f0846409013500454a11f6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260925 14:57:11.394004 6010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:11.395709 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f0f00189f0846409013500454a11f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0f00189f0846409013500454a11f6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } } }
I20260925 14:57:11.396011 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:11.397962 6022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f0f00189f0846409013500454a11f6a. Latest consensus state: current_term: 1 leader_uuid: "9f0f00189f0846409013500454a11f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0f00189f0846409013500454a11f6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } } }
I20260925 14:57:11.398265 6022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:11.401602 6026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:11.406315 6026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:11.414454 6026 catalog_manager.cc:1347] Generated new cluster ID: 2753bf812eac4f50ac229d8c5a07b8fb
I20260925 14:57:11.414544 6026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:11.435743 6026 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:11.437073 6026 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:11.456537 6026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f0f00189f0846409013500454a11f6a: Generated new TSK 0
I20260925 14:57:11.457569 6026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:11.474046 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 5943
W20260925 14:57:11.475247 6005 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53837
W20260925 14:57:11.476025 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 5943: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:11.476218 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 5943
W20260925 14:57:11.492797 5928 connection.cc:535] client connection to 127.0.0.1:41369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41369): Connection reset by peer (error 104)
2026-09-25T14:57:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:11 dist-test-slave-v5vl krb5kdc[6041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:11 dist-test-slave-v5vl krb5kdc[6041](info): set up 2 sockets
Sep 25 14:57:11 dist-test-slave-v5vl krb5kdc[6041](info): commencing operation
krb5kdc: starting...
W20260925 14:57:12.569052 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:12 dist-test-slave-v5vl krb5kdc[6041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:12.811276 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:13.130580 6054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:13.130878 6054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:13.131052 6054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:13.138253 6054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:13.138387 6054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:13.138509 6054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:13.138620 6054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:13.138706 6054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:13.148396 6054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:13.149745 6054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:13.151043 6054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:13.162686 6059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:13.162681 6060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:13.163161 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:13.163734 6054 server_base.cc:1034] running on GCE node
I20260925 14:57:13.164531 6054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:13.165956 6054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:13.167186 6054 hybrid_clock.cc:648] HybridClock initialized: now 1790348233167146 us; error 53 us; skew 500 ppm
I20260925 14:57:13.167547 6054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:13 dist-test-slave-v5vl krb5kdc[6041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:13.170764 6054 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:13.172149 6054 webserver.cc:460] Webserver started at http://127.0.0.1:45533/ using document root <none> and password file <none>
I20260925 14:57:13.172878 6054 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:13.173017 6054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:13.173317 6054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:13.176210 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "9955546d4ffd45bda52b03db2fa07fc4"
format_stamp: "Formatted at 2026-09-25 14:57:13 on dist-test-slave-v5vl"
server_key: "c575bb7722e45b16f0d4dce05783685e"
server_key_iv: "531bf6148feb360d477a1df357fbec55"
server_key_version: "encryptionkey@0"
I20260925 14:57:13.177002 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "9955546d4ffd45bda52b03db2fa07fc4"
format_stamp: "Formatted at 2026-09-25 14:57:13 on dist-test-slave-v5vl"
server_key: "c575bb7722e45b16f0d4dce05783685e"
server_key_iv: "531bf6148feb360d477a1df357fbec55"
server_key_version: "encryptionkey@0"
I20260925 14:57:13.181954 6054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 14:57:13.185207 6069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:13.186441 6054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:13.186612 6054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "9955546d4ffd45bda52b03db2fa07fc4"
format_stamp: "Formatted at 2026-09-25 14:57:13 on dist-test-slave-v5vl"
server_key: "c575bb7722e45b16f0d4dce05783685e"
server_key_iv: "531bf6148feb360d477a1df357fbec55"
server_key_version: "encryptionkey@0"
I20260925 14:57:13.186848 6054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:13.192366 6054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:13.196132 6054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:13.196425 6054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:13.214995 6054 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34755
I20260925 14:57:13.215035 6120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34755 every 8 connection(s)
I20260925 14:57:13.216462 6054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:13.220069 6121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:13.223714 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 6054
I20260925 14:57:13.223888 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:13.224399 17943 external_mini_cluster.cc:1428] Setting key ef5f915d08ce713cdafef6ca7da94274
I20260925 14:57:13.231879 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:13.226155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51953 (local address 127.0.0.1:34755)
0925 14:57:13.226921 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:13.226932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:13.226973 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:57:13.228688 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:13.228713 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:13.229065 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:13.229397 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:13.230421 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:13.230427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:13.230439 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:13.230455 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:13.230513 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:13.230534 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:13.230696 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:13.230824 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 14:57:13.231015 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":355,"threads_started":1}
I20260925 14:57:13.232658 6121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4: Bootstrap starting.
I20260925 14:57:13.235424 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:13.235948 6121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:13.236963 6121 log.cc:824] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:13.240065 6121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4: No bootstrap required, opened a new log
I20260925 14:57:13.242077 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:13.239759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32953 (local address 127.0.0.1:34755)
0925 14:57:13.239914 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:13.239919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:13.240293 (+ 374us) server_negotiation.cc:407] Connection header received
0925 14:57:13.240395 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:13.240400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:13.240463 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:13.240612 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:13.241435 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:13.241440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:13.241444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:13.241460 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:13.241505 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:13.241524 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:13.241660 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:13.241709 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 14:57:13.241769 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:57:13.247946 6121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9955546d4ffd45bda52b03db2fa07fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34755 } }
I20260925 14:57:13.248313 6121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:13.248377 6121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9955546d4ffd45bda52b03db2fa07fc4, State: Initialized, Role: FOLLOWER
I20260925 14:57:13.248983 6121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9955546d4ffd45bda52b03db2fa07fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34755 } }
I20260925 14:57:13.249279 6121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:13.249426 6121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:13.249647 6121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:13.251818 6121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9955546d4ffd45bda52b03db2fa07fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34755 } }
I20260925 14:57:13.252236 6121 leader_election.cc:302] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9955546d4ffd45bda52b03db2fa07fc4; no voters:
I20260925 14:57:13.252830 6121 leader_election.cc:288] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:13.252945 6131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:13.253643 6131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [term 1 LEADER]: Becoming Leader. State: Replica: 9955546d4ffd45bda52b03db2fa07fc4, State: Running, Role: LEADER
I20260925 14:57:13.254189 6131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9955546d4ffd45bda52b03db2fa07fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34755 } }
I20260925 14:57:13.255102 6121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:13.257118 6133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9955546d4ffd45bda52b03db2fa07fc4. Latest consensus state: current_term: 1 leader_uuid: "9955546d4ffd45bda52b03db2fa07fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9955546d4ffd45bda52b03db2fa07fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34755 } } }
I20260925 14:57:13.257560 6133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:13.257588 6132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9955546d4ffd45bda52b03db2fa07fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9955546d4ffd45bda52b03db2fa07fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34755 } } }
I20260925 14:57:13.258414 6132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:13.260070 6140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:13.264554 6140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:13.272112 6140 catalog_manager.cc:1347] Generated new cluster ID: 798d87b29b004eeba4b817adad648470
I20260925 14:57:13.272222 6140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:13.295156 6140 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:13.295687 6140 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:13.308358 6140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9955546d4ffd45bda52b03db2fa07fc4: Generated new TSK 0
I20260925 14:57:13.309345 6140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:13.314160 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6054
W20260925 14:57:13.315796 6116 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51953
W20260925 14:57:13.316573 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6054: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:13.316749 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 6054
2026-09-25T14:57:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1840 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:13 dist-test-slave-v5vl krb5kdc[6152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:13 dist-test-slave-v5vl krb5kdc[6152](info): set up 2 sockets
Sep 25 14:57:13 dist-test-slave-v5vl krb5kdc[6152](info): commencing operation
krb5kdc: starting...
W20260925 14:57:14.409194 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.014s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:14 dist-test-slave-v5vl krb5kdc[6152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:14.655280 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:14.979862 6165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:14.980214 6165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:14.980378 6165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:14.987699 6165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:14.987849 6165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:14.987943 6165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:14.988047 6165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:14.988132 6165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:14.998013 6165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:14.999265 6165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:15.000562 6165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:15.012481 6170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:15.012516 6171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:15.012552 6173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:15.013705 6165 server_base.cc:1034] running on GCE node
I20260925 14:57:15.014479 6165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:15.015853 6165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:15.017081 6165 hybrid_clock.cc:648] HybridClock initialized: now 1790348235017033 us; error 55 us; skew 500 ppm
I20260925 14:57:15.017426 6165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:15 dist-test-slave-v5vl krb5kdc[6152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:15.026189 6165 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:15.027766 6165 webserver.cc:460] Webserver started at http://127.0.0.1:39619/ using document root <none> and password file <none>
I20260925 14:57:15.028415 6165 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:15.028529 6165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:15.028856 6165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:15.031824 6165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "3635b25d6700412b840ca6e747b96423"
format_stamp: "Formatted at 2026-09-25 14:57:15 on dist-test-slave-v5vl"
server_key: "15769d86e703054c0271a37702908191"
server_key_iv: "43a2b5babbd86c094a0f94145c84bf2d"
server_key_version: "encryptionkey@0"
I20260925 14:57:15.032552 6165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "3635b25d6700412b840ca6e747b96423"
format_stamp: "Formatted at 2026-09-25 14:57:15 on dist-test-slave-v5vl"
server_key: "15769d86e703054c0271a37702908191"
server_key_iv: "43a2b5babbd86c094a0f94145c84bf2d"
server_key_version: "encryptionkey@0"
I20260925 14:57:15.037786 6165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:57:15.041230 6180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:15.042512 6165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:57:15.042697 6165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "3635b25d6700412b840ca6e747b96423"
format_stamp: "Formatted at 2026-09-25 14:57:15 on dist-test-slave-v5vl"
server_key: "15769d86e703054c0271a37702908191"
server_key_iv: "43a2b5babbd86c094a0f94145c84bf2d"
server_key_version: "encryptionkey@0"
I20260925 14:57:15.042891 6165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:15.058535 6165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:15.062232 6165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:15.062553 6165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:15.080852 6165 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33239
I20260925 14:57:15.080883 6231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33239 every 8 connection(s)
I20260925 14:57:15.082203 6165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:15.086062 6232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:15.089506 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 6165
I20260925 14:57:15.089673 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:15.090112 17943 external_mini_cluster.cc:1428] Setting key 3f5cb7accd292f66285b895d28baabbb
I20260925 14:57:15.098327 6232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423: Bootstrap starting.
I20260925 14:57:15.102084 6232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:15.103411 6232 log.cc:824] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423: Log is configured to *not* fsync() on all Append() calls
Sep 25 14:57:15 dist-test-slave-v5vl krb5kdc[6152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:57:15.106297 6232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423: No bootstrap required, opened a new log
I20260925 14:57:15.112651 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:15.091883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40945 (local address 127.0.0.1:33239)
0925 14:57:15.092735 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:15.092747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:15.092785 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:57:15.094302 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:15.094323 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:15.094718 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:15.095023 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:15.096781 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:15.098067 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:15.099778 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:15.100439 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:15.100563 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:15.105609 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:15.105617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:15.105630 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:15.105670 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:15.108767 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:15.109454 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:15.109460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:15.109466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:15.109670 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:15.110068 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:15.110075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:15.110083 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:15.110569 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:15.110902 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:15.111268 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 14:57:15.111664 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":379,"threads_started":1}
I20260925 14:57:15.114192 6232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635b25d6700412b840ca6e747b96423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33239 } }
I20260925 14:57:15.114530 6232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:15.114655 6232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3635b25d6700412b840ca6e747b96423, State: Initialized, Role: FOLLOWER
I20260925 14:57:15.115298 6232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635b25d6700412b840ca6e747b96423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33239 } }
I20260925 14:57:15.115574 6232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:15.115717 6232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:15.115890 6232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:15.116261 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:15.118206 6232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635b25d6700412b840ca6e747b96423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33239 } }
I20260925 14:57:15.118782 6232 leader_election.cc:302] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3635b25d6700412b840ca6e747b96423; no voters:
I20260925 14:57:15.119447 6232 leader_election.cc:288] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:15.119598 6239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:15.121971 6232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:15.126228 6239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [term 1 LEADER]: Becoming Leader. State: Replica: 3635b25d6700412b840ca6e747b96423, State: Running, Role: LEADER
I20260925 14:57:15.127032 6239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635b25d6700412b840ca6e747b96423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33239 } }
I20260925 14:57:15.131152 6242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3635b25d6700412b840ca6e747b96423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635b25d6700412b840ca6e747b96423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33239 } } }
I20260925 14:57:15.131606 6242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:15.132102 6243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3635b25d6700412b840ca6e747b96423. Latest consensus state: current_term: 1 leader_uuid: "3635b25d6700412b840ca6e747b96423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635b25d6700412b840ca6e747b96423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33239 } } }
I20260925 14:57:15.132345 6243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:15.136356 6255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:15.142884 6255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:15.143162 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:15.120820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38221 (local address 127.0.0.1:33239)
0925 14:57:15.121014 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:15.121023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:15.125563 (+ 4540us) server_negotiation.cc:407] Connection header received
0925 14:57:15.126028 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:15.126036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:15.126140 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:15.126234 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:15.127925 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:15.128804 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:15.130787 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:15.131624 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:15.131800 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:15.135747 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:15.135756 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:15.135764 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:15.135809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:15.139939 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:15.140749 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:15.140757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:15.140761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:15.140931 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:15.141322 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:15.141328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:15.141332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:15.141753 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:15.142030 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:15.142374 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 14:57:15.142600 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:57:15.159300 6255 catalog_manager.cc:1347] Generated new cluster ID: e7d289bd75b14624b35819b2ec0266c2
I20260925 14:57:15.159435 6255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:15.174340 6255 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:15.176095 6255 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:15.194538 6255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3635b25d6700412b840ca6e747b96423: Generated new TSK 0
I20260925 14:57:15.195581 6255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:15.212981 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6165
I20260925 14:57:15.213943 6227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:15.307011 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 6165
2026-09-25T14:57:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:15 dist-test-slave-v5vl krb5kdc[6264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:15 dist-test-slave-v5vl krb5kdc[6264](info): set up 2 sockets
Sep 25 14:57:15 dist-test-slave-v5vl krb5kdc[6264](info): commencing operation
krb5kdc: starting...
W20260925 14:57:16.401086 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:16 dist-test-slave-v5vl krb5kdc[6264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:16.645491 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:16.963910 6277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:16.964260 6277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:16.964442 6277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:16.971757 6277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:16.971907 6277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:16.972021 6277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:16.972132 6277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:16.972198 6277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:16.981745 6277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:16.983050 6277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:16.984354 6277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:16.995831 6283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:16.995895 6282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:16.995918 6285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:16.996881 6277 server_base.cc:1034] running on GCE node
I20260925 14:57:16.997691 6277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:16.999104 6277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:17.000387 6277 hybrid_clock.cc:648] HybridClock initialized: now 1790348237000334 us; error 48 us; skew 500 ppm
I20260925 14:57:17.000784 6277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:16 dist-test-slave-v5vl krb5kdc[6264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:17.003808 6277 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:17.005280 6277 webserver.cc:460] Webserver started at http://127.0.0.1:36879/ using document root <none> and password file <none>
I20260925 14:57:17.005882 6277 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:17.005990 6277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:17.006237 6277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:17.009248 6277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5"
format_stamp: "Formatted at 2026-09-25 14:57:17 on dist-test-slave-v5vl"
server_key: "65d38d80257013807203a1ac85dd39ad"
server_key_iv: "267381868fd39bcebd86380ddc4030ef"
server_key_version: "encryptionkey@0"
I20260925 14:57:17.009945 6277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5"
format_stamp: "Formatted at 2026-09-25 14:57:17 on dist-test-slave-v5vl"
server_key: "65d38d80257013807203a1ac85dd39ad"
server_key_iv: "267381868fd39bcebd86380ddc4030ef"
server_key_version: "encryptionkey@0"
I20260925 14:57:17.015164 6277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:57:17.019357 6292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:17.020951 6277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:57:17.021145 6277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5"
format_stamp: "Formatted at 2026-09-25 14:57:17 on dist-test-slave-v5vl"
server_key: "65d38d80257013807203a1ac85dd39ad"
server_key_iv: "267381868fd39bcebd86380ddc4030ef"
server_key_version: "encryptionkey@0"
I20260925 14:57:17.021315 6277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:17.046761 6277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:17.051309 6277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:17.051609 6277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:17.069686 6277 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32807
I20260925 14:57:17.069720 6343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32807 every 8 connection(s)
I20260925 14:57:17.071071 6277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:17.074712 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:17.080817 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 6277
I20260925 14:57:17.080999 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:17.081507 17943 external_mini_cluster.cc:1428] Setting key 4ff9a7aa0f5a39aa58298b86aff71387
I20260925 14:57:17.085728 6344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5: Bootstrap starting.
I20260925 14:57:17.089392 6344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:17.090632 6344 log.cc:824] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:17.093950 6344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5: No bootstrap required, opened a new log
I20260925 14:57:17.095340 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:17.083357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35639 (local address 127.0.0.1:32807)
0925 14:57:17.084138 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:17.084149 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:17.084188 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:57:17.085957 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:17.085990 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:17.086369 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:17.086680 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:17.087254 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:17.088781 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:17.090712 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:17.091784 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:17.091994 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:17.093141 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:17.093148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:17.093162 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:17.093666 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:17.093737 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:17.093761 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:17.093847 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:17.094138 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 14:57:17.094427 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":387,"threads_started":1}
I20260925 14:57:17.098956 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:17.102082 6344 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } }
I20260925 14:57:17.102459 6344 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:17.102555 6344 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1c5b2bc85e9426b8e7fbacc8dc706d5, State: Initialized, Role: FOLLOWER
I20260925 14:57:17.103260 6344 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } }
I20260925 14:57:17.103539 6344 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:17.103709 6344 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:17.103883 6344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:17.106761 6344 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } }
I20260925 14:57:17.107306 6344 leader_election.cc:302] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1c5b2bc85e9426b8e7fbacc8dc706d5; no voters:
I20260925 14:57:17.107896 6344 leader_election.cc:288] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:17.108028 6354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:17.108668 6354 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [term 1 LEADER]: Becoming Leader. State: Replica: f1c5b2bc85e9426b8e7fbacc8dc706d5, State: Running, Role: LEADER
I20260925 14:57:17.109313 6354 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } }
I20260925 14:57:17.112079 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:17.103317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60621 (local address 127.0.0.1:32807)
0925 14:57:17.103527 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:17.103535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:17.105112 (+ 1577us) server_negotiation.cc:407] Connection header received
0925 14:57:17.105231 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:17.105237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:17.105319 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:17.105438 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:17.105915 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:17.106978 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:17.108803 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:17.109485 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:17.109579 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:17.111123 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:17.111130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:17.111134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:17.111153 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:17.111201 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:17.111220 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:17.111293 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:17.111557 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 14:57:17.111670 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:57:17.111972 6356 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } } }
I20260925 14:57:17.112373 6356 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:17.114681 6355 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1c5b2bc85e9426b8e7fbacc8dc706d5. Latest consensus state: current_term: 1 leader_uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c5b2bc85e9426b8e7fbacc8dc706d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } } }
I20260925 14:57:17.115043 6355 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:17.117254 6344 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:17.119146 6358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:17.125833 6358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:17.135048 6358 catalog_manager.cc:1347] Generated new cluster ID: 37123354f22b4e5ba3a6afecb56b38a7
I20260925 14:57:17.135190 6358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:17.155043 6358 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:17.156369 6358 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:17.161852 6358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1c5b2bc85e9426b8e7fbacc8dc706d5: Generated new TSK 0
I20260925 14:57:17.162640 6358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:17.170421 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6277
W20260925 14:57:17.171434 6339 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35639
W20260925 14:57:17.172075 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6277: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:17.172225 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 6277
2026-09-25T14:57:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1864 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:17 dist-test-slave-v5vl krb5kdc[6376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:17 dist-test-slave-v5vl krb5kdc[6376](info): set up 2 sockets
Sep 25 14:57:17 dist-test-slave-v5vl krb5kdc[6376](info): commencing operation
krb5kdc: starting...
W20260925 14:57:18.261233 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:18 dist-test-slave-v5vl krb5kdc[6376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:18.504488 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:18.828600 6389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:18.828933 6389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:18.829082 6389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:18.836345 6389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:18.836458 6389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:18.836505 6389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:18.836575 6389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:18.836683 6389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:18.846643 6389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:18.847909 6389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:18.849164 6389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:18.860634 6397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:18.860697 6395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:18.860915 6394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:18.861690 6389 server_base.cc:1034] running on GCE node
I20260925 14:57:18.862504 6389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:18.863925 6389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:18.865236 6389 hybrid_clock.cc:648] HybridClock initialized: now 1790348238865216 us; error 61 us; skew 500 ppm
I20260925 14:57:18.865638 6389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:18 dist-test-slave-v5vl krb5kdc[6376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:18.868845 6389 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:18.870229 6389 webserver.cc:460] Webserver started at http://127.0.0.1:37703/ using document root <none> and password file <none>
I20260925 14:57:18.870836 6389 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:18.870934 6389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:18.871170 6389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:18.874114 6389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "7b126b5cbad04d2c873c7710d4c817d1"
format_stamp: "Formatted at 2026-09-25 14:57:18 on dist-test-slave-v5vl"
server_key: "bd2a06208a62d0a62e754b50f13478dc"
server_key_iv: "c1bc65e0659a9df3e73ff1900fd09d3b"
server_key_version: "encryptionkey@0"
I20260925 14:57:18.874845 6389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "7b126b5cbad04d2c873c7710d4c817d1"
format_stamp: "Formatted at 2026-09-25 14:57:18 on dist-test-slave-v5vl"
server_key: "bd2a06208a62d0a62e754b50f13478dc"
server_key_iv: "c1bc65e0659a9df3e73ff1900fd09d3b"
server_key_version: "encryptionkey@0"
I20260925 14:57:18.880018 6389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:57:18.883617 6404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:18.885169 6389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:18.885363 6389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "7b126b5cbad04d2c873c7710d4c817d1"
format_stamp: "Formatted at 2026-09-25 14:57:18 on dist-test-slave-v5vl"
server_key: "bd2a06208a62d0a62e754b50f13478dc"
server_key_iv: "c1bc65e0659a9df3e73ff1900fd09d3b"
server_key_version: "encryptionkey@0"
I20260925 14:57:18.885587 6389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:18.891446 6389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:18.895881 6389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:18.896198 6389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:18.915552 6389 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35359
I20260925 14:57:18.915587 6455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35359 every 8 connection(s)
I20260925 14:57:18.917021 6389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:18.917393 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 6389
I20260925 14:57:18.917557 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:18.918139 17943 external_mini_cluster.cc:1428] Setting key 97002c0aa048fa8c045f617adb1e52f6
I20260925 14:57:18.921561 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:18.926934 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:18.920012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51421 (local address 127.0.0.1:35359)
0925 14:57:18.921124 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:18.921147 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:18.921729 (+ 582us) server_negotiation.cc:407] Connection header received
0925 14:57:18.923502 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:18.923523 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:18.923871 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:18.924267 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:18.925322 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:18.925338 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:18.925351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:18.925365 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:18.925440 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:18.925463 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:18.925710 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:18.925867 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 14:57:18.926048 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":481,"threads_started":1}
I20260925 14:57:18.930585 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:18.933046 6456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1: Bootstrap starting.
I20260925 14:57:18.936488 6456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:18.937701 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:18.934794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34155 (local address 127.0.0.1:35359)
0925 14:57:18.935014 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:18.935023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:18.935399 (+ 376us) server_negotiation.cc:407] Connection header received
0925 14:57:18.935584 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:18.935588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:18.935634 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:18.935711 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:18.936874 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:18.936886 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:18.936892 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:18.936916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:18.936973 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:18.937002 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:18.937093 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:18.937343 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 14:57:18.937405 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:57:18.937978 6456 log.cc:824] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:18.940876 6456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1: No bootstrap required, opened a new log
I20260925 14:57:18.947897 6456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b126b5cbad04d2c873c7710d4c817d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } }
I20260925 14:57:18.948282 6456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:18.948382 6456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b126b5cbad04d2c873c7710d4c817d1, State: Initialized, Role: FOLLOWER
I20260925 14:57:18.949023 6456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b126b5cbad04d2c873c7710d4c817d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } }
I20260925 14:57:18.949254 6456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:18.949393 6456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:18.949559 6456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:18.951800 6456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b126b5cbad04d2c873c7710d4c817d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } }
I20260925 14:57:18.952260 6456 leader_election.cc:302] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b126b5cbad04d2c873c7710d4c817d1; no voters:
I20260925 14:57:18.952903 6456 leader_election.cc:288] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:18.953029 6466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:18.953884 6466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [term 1 LEADER]: Becoming Leader. State: Replica: 7b126b5cbad04d2c873c7710d4c817d1, State: Running, Role: LEADER
I20260925 14:57:18.954440 6466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b126b5cbad04d2c873c7710d4c817d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } }
I20260925 14:57:18.954910 6456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:18.957391 6468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b126b5cbad04d2c873c7710d4c817d1. Latest consensus state: current_term: 1 leader_uuid: "7b126b5cbad04d2c873c7710d4c817d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b126b5cbad04d2c873c7710d4c817d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } } }
I20260925 14:57:18.957351 6467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b126b5cbad04d2c873c7710d4c817d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b126b5cbad04d2c873c7710d4c817d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35359 } } }
I20260925 14:57:18.957796 6468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:18.957825 6467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:18.959385 6473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:18.964545 6473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:18.973564 6473 catalog_manager.cc:1347] Generated new cluster ID: 62b788beba5341629fd52651442199a0
I20260925 14:57:18.973683 6473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:18.995611 6473 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:18.996049 6473 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:19.002562 6473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b126b5cbad04d2c873c7710d4c817d1: Generated new TSK 0
I20260925 14:57:19.003554 6473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:19.008811 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6389
W20260925 14:57:19.010927 6451 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51421
W20260925 14:57:19.011683 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:19.011833 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 6389
2026-09-25T14:57:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1839 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:19 dist-test-slave-v5vl krb5kdc[6487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:19 dist-test-slave-v5vl krb5kdc[6487](info): set up 2 sockets
Sep 25 14:57:19 dist-test-slave-v5vl krb5kdc[6487](info): commencing operation
krb5kdc: starting...
W20260925 14:57:20.109091 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:20 dist-test-slave-v5vl krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:20.368341 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:20.693452 6500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:20.693792 6500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:20.693950 6500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:20.701336 6500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:20.701491 6500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:20.701604 6500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:20.701713 6500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:20.701802 6500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:20.711717 6500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:20.713104 6500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:20.714395 6500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:20.726063 6505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:20.726007 6508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:20.726099 6506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:20.726835 6500 server_base.cc:1034] running on GCE node
I20260925 14:57:20.727615 6500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:20.729085 6500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:20.730310 6500 hybrid_clock.cc:648] HybridClock initialized: now 1790348240730267 us; error 53 us; skew 500 ppm
I20260925 14:57:20.730666 6500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:20 dist-test-slave-v5vl krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:20.733719 6500 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:20.735122 6500 webserver.cc:460] Webserver started at http://127.0.0.1:36925/ using document root <none> and password file <none>
I20260925 14:57:20.735721 6500 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:20.735816 6500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:20.736050 6500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:20.738976 6500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "6af0f9cba808420b829338615977e6ca"
format_stamp: "Formatted at 2026-09-25 14:57:20 on dist-test-slave-v5vl"
server_key: "7066bb433b1def505e1d8074557a3391"
server_key_iv: "24e977e1c70d81f42294a3403ffc894b"
server_key_version: "encryptionkey@0"
I20260925 14:57:20.739681 6500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "6af0f9cba808420b829338615977e6ca"
format_stamp: "Formatted at 2026-09-25 14:57:20 on dist-test-slave-v5vl"
server_key: "7066bb433b1def505e1d8074557a3391"
server_key_iv: "24e977e1c70d81f42294a3403ffc894b"
server_key_version: "encryptionkey@0"
I20260925 14:57:20.744542 6500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 14:57:20.747823 6515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:20.749235 6500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:20.749410 6500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "6af0f9cba808420b829338615977e6ca"
format_stamp: "Formatted at 2026-09-25 14:57:20 on dist-test-slave-v5vl"
server_key: "7066bb433b1def505e1d8074557a3391"
server_key_iv: "24e977e1c70d81f42294a3403ffc894b"
server_key_version: "encryptionkey@0"
I20260925 14:57:20.749570 6500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:20.761209 6500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:20.765759 6500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:20.766058 6500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:20.784308 6500 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44807
I20260925 14:57:20.784345 6566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44807 every 8 connection(s)
I20260925 14:57:20.785688 6500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:20.789417 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:20.792797 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 6500
I20260925 14:57:20.792989 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:20.793529 17943 external_mini_cluster.cc:1428] Setting key 5a4c91691137c57a7437aa5e7f5019bb
I20260925 14:57:20.801761 6567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca: Bootstrap starting.
I20260925 14:57:20.805557 6567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:20.806917 6567 log.cc:824] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca: Log is configured to *not* fsync() on all Append() calls
Sep 25 14:57:20 dist-test-slave-v5vl krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:57:20.810032 6567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca: No bootstrap required, opened a new log
I20260925 14:57:20.816576 6570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:20.795263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:56909 (local address 127.0.0.1:44807)
0925 14:57:20.796022 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:20.796034 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:20.796070 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:57:20.797734 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:20.797754 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:20.798147 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:20.798472 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:20.800189 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:20.801536 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:20.803271 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:20.803943 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:20.804068 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:20.809257 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:20.809265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:20.809278 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:20.809312 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:20.812435 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:20.813402 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:20.813408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:20.813414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:20.813578 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:20.814044 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:20.814047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:20.814049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:20.814516 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:20.814748 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:20.815235 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 14:57:20.815553 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":326,"threads_started":1}
I20260925 14:57:20.819296 6567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af0f9cba808420b829338615977e6ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260925 14:57:20.819672 6567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:20.819806 6567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0f9cba808420b829338615977e6ca, State: Initialized, Role: FOLLOWER
I20260925 14:57:20.820329 6567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af0f9cba808420b829338615977e6ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260925 14:57:20.820567 6567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:20.820631 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:20.820755 6567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:20.820930 6567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:20.823047 6567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af0f9cba808420b829338615977e6ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260925 14:57:20.823609 6567 leader_election.cc:302] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6af0f9cba808420b829338615977e6ca; no voters:
I20260925 14:57:20.824331 6567 leader_election.cc:288] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:20.824440 6576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:20.830325 6576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [term 1 LEADER]: Becoming Leader. State: Replica: 6af0f9cba808420b829338615977e6ca, State: Running, Role: LEADER
I20260925 14:57:20.830964 6576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af0f9cba808420b829338615977e6ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260925 14:57:20.831346 6567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:20.833717 6579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6af0f9cba808420b829338615977e6ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af0f9cba808420b829338615977e6ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260925 14:57:20.834092 6579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:20.834718 6577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6af0f9cba808420b829338615977e6ca. Latest consensus state: current_term: 1 leader_uuid: "6af0f9cba808420b829338615977e6ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af0f9cba808420b829338615977e6ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260925 14:57:20.835070 6577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:20.838423 6584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:20.845184 6584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:20.846479 6570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:20.824972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:36547 (local address 127.0.0.1:44807)
0925 14:57:20.825212 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:20.825221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:20.829129 (+ 3908us) server_negotiation.cc:407] Connection header received
0925 14:57:20.829319 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:20.829326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:20.829423 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:20.829534 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:20.831365 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:20.832218 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:20.834167 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:20.834992 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:20.835125 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:20.838823 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:20.838830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:20.838834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:20.838863 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:20.842356 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:20.843387 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:20.843394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:20.843399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:20.843554 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:20.844105 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:20.844109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:20.844112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:20.844553 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:20.845720 (+ 1167us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:20.845895 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 14:57:20.846079 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:57:20.857177 6584 catalog_manager.cc:1347] Generated new cluster ID: e0a67b9614814393881d475d9894fca1
I20260925 14:57:20.857311 6584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:20.874374 6584 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:20.875624 6584 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:20.883811 6584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6af0f9cba808420b829338615977e6ca: Generated new TSK 0
I20260925 14:57:20.884568 6584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:20.890974 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6500
I20260925 14:57:20.892537 6562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:20.983235 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 6500
W20260925 14:57:21.000303 6485 connection.cc:535] client connection to 127.0.0.1:44807 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44807): Connection reset by peer (error 104)
2026-09-25T14:57:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:21 dist-test-slave-v5vl krb5kdc[6599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:21 dist-test-slave-v5vl krb5kdc[6599](info): set up 2 sockets
Sep 25 14:57:21 dist-test-slave-v5vl krb5kdc[6599](info): commencing operation
krb5kdc: starting...
W20260925 14:57:22.085151 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:22 dist-test-slave-v5vl krb5kdc[6599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:22.335459 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:22.664561 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:22.664984 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:22.665165 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:22.672400 6612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:22.672516 6612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:22.672662 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:22.672766 6612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:22.672865 6612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:22.682960 6612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:22.684320 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:22.685701 6612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:22.697736 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:22.697685 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:22.697710 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:22.699275 6612 server_base.cc:1034] running on GCE node
I20260925 14:57:22.700065 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:22.701613 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:22.702814 6612 hybrid_clock.cc:648] HybridClock initialized: now 1790348242702752 us; error 55 us; skew 500 ppm
I20260925 14:57:22.703154 6612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:22 dist-test-slave-v5vl krb5kdc[6599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:22.706450 6612 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:22.707924 6612 webserver.cc:460] Webserver started at http://127.0.0.1:33097/ using document root <none> and password file <none>
I20260925 14:57:22.708666 6612 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:22.708791 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:22.709062 6612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:22.712033 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "5b9ba9250d354abfb1065a1f848485c5"
format_stamp: "Formatted at 2026-09-25 14:57:22 on dist-test-slave-v5vl"
server_key: "54a97244e4073a4a8f81ddd4d445b084"
server_key_iv: "020296fe27f791c10401824da267df67"
server_key_version: "encryptionkey@0"
I20260925 14:57:22.712837 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "5b9ba9250d354abfb1065a1f848485c5"
format_stamp: "Formatted at 2026-09-25 14:57:22 on dist-test-slave-v5vl"
server_key: "54a97244e4073a4a8f81ddd4d445b084"
server_key_iv: "020296fe27f791c10401824da267df67"
server_key_version: "encryptionkey@0"
I20260925 14:57:22.718146 6612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 14:57:22.721813 6627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:22.723181 6612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:57:22.723409 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "5b9ba9250d354abfb1065a1f848485c5"
format_stamp: "Formatted at 2026-09-25 14:57:22 on dist-test-slave-v5vl"
server_key: "54a97244e4073a4a8f81ddd4d445b084"
server_key_iv: "020296fe27f791c10401824da267df67"
server_key_version: "encryptionkey@0"
I20260925 14:57:22.723623 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:22.734965 6612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:22.739336 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:22.739619 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:22.758198 6612 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39573
I20260925 14:57:22.758219 6678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39573 every 8 connection(s)
I20260925 14:57:22.759680 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:22.763393 6680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:22.768900 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 6612
I20260925 14:57:22.769100 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:22.769518 17943 external_mini_cluster.cc:1428] Setting key 7e83586ece2d1060a5abf7fefe6f9aae
I20260925 14:57:22.774428 6680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5: Bootstrap starting.
I20260925 14:57:22.777529 6680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:22.778545 6680 log.cc:824] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:22.781780 6680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5: No bootstrap required, opened a new log
I20260925 14:57:22.782742 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:22.771300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:53561 (local address 127.0.0.1:39573)
0925 14:57:22.772303 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:22.772315 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:22.772356 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:57:22.774063 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:22.774088 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:22.774493 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:22.774910 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:22.775439 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:22.776838 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:22.778228 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:22.779007 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:22.779208 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:22.780458 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:22.780466 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:22.780479 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:22.781082 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:22.781153 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:22.781177 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:22.781299 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:22.781567 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 14:57:22.781757 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":511,"threads_started":1}
I20260925 14:57:22.786680 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:22.791997 6680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9ba9250d354abfb1065a1f848485c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } }
I20260925 14:57:22.792377 6680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:22.792510 6680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9ba9250d354abfb1065a1f848485c5, State: Initialized, Role: FOLLOWER
I20260925 14:57:22.793157 6680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9ba9250d354abfb1065a1f848485c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } }
I20260925 14:57:22.793407 6680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:22.793589 6680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:22.793781 6680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:22.796031 6680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9ba9250d354abfb1065a1f848485c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } }
I20260925 14:57:22.796569 6680 leader_election.cc:302] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b9ba9250d354abfb1065a1f848485c5; no voters:
I20260925 14:57:22.797343 6680 leader_election.cc:288] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:22.797492 6691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:22.798548 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:22.791688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:56515 (local address 127.0.0.1:39573)
0925 14:57:22.791872 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:22.791878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:22.792459 (+ 581us) server_negotiation.cc:407] Connection header received
0925 14:57:22.792683 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:22.792687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:22.792748 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:22.792861 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:22.793371 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:22.794293 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:22.795626 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:22.796121 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:22.796264 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:22.797553 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:22.797560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:22.797564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:22.797587 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:22.797633 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:22.797663 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:22.797841 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:22.798170 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 14:57:22.798223 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:57:22.798874 6691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [term 1 LEADER]: Becoming Leader. State: Replica: 5b9ba9250d354abfb1065a1f848485c5, State: Running, Role: LEADER
I20260925 14:57:22.799809 6680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:22.799774 6691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9ba9250d354abfb1065a1f848485c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } }
I20260925 14:57:22.802938 6693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b9ba9250d354abfb1065a1f848485c5. Latest consensus state: current_term: 1 leader_uuid: "5b9ba9250d354abfb1065a1f848485c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9ba9250d354abfb1065a1f848485c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } } }
I20260925 14:57:22.803047 6692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b9ba9250d354abfb1065a1f848485c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9ba9250d354abfb1065a1f848485c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } } }
I20260925 14:57:22.803334 6693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:22.803421 6692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:22.805794 6698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:22.811316 6698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:22.820487 6698 catalog_manager.cc:1347] Generated new cluster ID: 58eadb6cda3d4418bf0227958be706e8
I20260925 14:57:22.820640 6698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:22.834146 6698 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:22.835508 6698 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:22.841485 6698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b9ba9250d354abfb1065a1f848485c5: Generated new TSK 0
I20260925 14:57:22.842324 6698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:22.848752 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6612
W20260925 14:57:22.849920 6674 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.57:53561
W20260925 14:57:22.850708 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6612: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:22.850865 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 6612
W20260925 14:57:22.867863 6597 connection.cc:535] client connection to 127.0.0.1:39573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39573): Connection reset by peer (error 104)
2026-09-25T14:57:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:22 dist-test-slave-v5vl krb5kdc[6712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:22 dist-test-slave-v5vl krb5kdc[6712](info): set up 2 sockets
Sep 25 14:57:22 dist-test-slave-v5vl krb5kdc[6712](info): commencing operation
krb5kdc: starting...
W20260925 14:57:23.953141 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:24 dist-test-slave-v5vl krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:24.215930 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:24.543668 6725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:24.543963 6725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:24.544052 6725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:24.551441 6725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:24.551566 6725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:24.551613 6725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:24.551698 6725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:24.551767 6725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:24.561933 6725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:24.563302 6725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:24.564661 6725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:24.576716 6730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:24.576740 6733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:24.577150 6731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:24.578015 6725 server_base.cc:1034] running on GCE node
I20260925 14:57:24.578858 6725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:24.580220 6725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:24.581427 6725 hybrid_clock.cc:648] HybridClock initialized: now 1790348244581388 us; error 25 us; skew 500 ppm
I20260925 14:57:24.581753 6725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:24 dist-test-slave-v5vl krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:24.584789 6725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:24.586251 6725 webserver.cc:460] Webserver started at http://127.0.0.1:40501/ using document root <none> and password file <none>
I20260925 14:57:24.586861 6725 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:24.586958 6725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:24.587212 6725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:24.590128 6725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "e49f7aff3df049b19d150ecb11e4b171"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-v5vl"
server_key: "b926569b63620c77b3aa667484d2ed9f"
server_key_iv: "9417966756498497de3c32967bcb37c8"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.590838 6725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "e49f7aff3df049b19d150ecb11e4b171"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-v5vl"
server_key: "b926569b63620c77b3aa667484d2ed9f"
server_key_iv: "9417966756498497de3c32967bcb37c8"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.595992 6725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 14:57:24.599591 6740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.601209 6725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:24.601406 6725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "e49f7aff3df049b19d150ecb11e4b171"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-v5vl"
server_key: "b926569b63620c77b3aa667484d2ed9f"
server_key_iv: "9417966756498497de3c32967bcb37c8"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.601625 6725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:24.607589 6725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:24.612032 6725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:24.612335 6725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:24.633037 6725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42517
I20260925 14:57:24.633069 6792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42517 every 8 connection(s)
I20260925 14:57:24.634455 6725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:24.638053 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:24.642021 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 6725
I20260925 14:57:24.642200 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:24.642717 17943 external_mini_cluster.cc:1428] Setting key 930c7cb14948265d99804c5eaef8c7b5
I20260925 14:57:24.650538 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.644607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:60831 (local address 127.0.0.1:42517)
0925 14:57:24.645377 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.645388 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.645429 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:57:24.647058 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.647079 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.647459 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.647873 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:24.648972 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:24.648978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:24.648991 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:24.649004 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:24.649057 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:24.649075 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:24.649319 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.649455 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.649661 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":312,"threads_started":1}
I20260925 14:57:24.650785 6793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171: Bootstrap starting.
I20260925 14:57:24.653980 6793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:24.654018 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:24.654973 6793 log.cc:824] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:24.657915 6793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171: No bootstrap required, opened a new log
I20260925 14:57:24.661154 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.658275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:39245 (local address 127.0.0.1:42517)
0925 14:57:24.658459 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.658464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.659080 (+ 616us) server_negotiation.cc:407] Connection header received
0925 14:57:24.659282 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.659286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.659329 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.659431 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:24.660464 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:24.660469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:24.660472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:24.660486 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:24.660523 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:24.660537 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:24.660694 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.660816 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.660878 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 14:57:24.666531 6793 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f7aff3df049b19d150ecb11e4b171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42517 } }
I20260925 14:57:24.666980 6793 raft_consensus.cc:374] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:24.667095 6793 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49f7aff3df049b19d150ecb11e4b171, State: Initialized, Role: FOLLOWER
I20260925 14:57:24.667660 6793 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f7aff3df049b19d150ecb11e4b171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42517 } }
I20260925 14:57:24.667950 6793 raft_consensus.cc:388] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:24.668156 6793 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:24.668352 6793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:24.670559 6793 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f7aff3df049b19d150ecb11e4b171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42517 } }
I20260925 14:57:24.671032 6793 leader_election.cc:302] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49f7aff3df049b19d150ecb11e4b171; no voters:
I20260925 14:57:24.671597 6793 leader_election.cc:288] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:24.671725 6803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:24.672345 6803 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [term 1 LEADER]: Becoming Leader. State: Replica: e49f7aff3df049b19d150ecb11e4b171, State: Running, Role: LEADER
I20260925 14:57:24.672904 6803 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f7aff3df049b19d150ecb11e4b171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42517 } }
I20260925 14:57:24.673682 6793 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:24.675650 6805 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49f7aff3df049b19d150ecb11e4b171. Latest consensus state: current_term: 1 leader_uuid: "e49f7aff3df049b19d150ecb11e4b171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f7aff3df049b19d150ecb11e4b171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42517 } } }
I20260925 14:57:24.676118 6805 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:24.675676 6804 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49f7aff3df049b19d150ecb11e4b171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49f7aff3df049b19d150ecb11e4b171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42517 } } }
I20260925 14:57:24.676693 6804 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:24.678869 6808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:24.684000 6808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:24.692948 6808 catalog_manager.cc:1347] Generated new cluster ID: be546c0a60b54111ab3eb962b5cb54d0
I20260925 14:57:24.693073 6808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:24.707623 6808 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:24.708076 6808 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:24.714236 6808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e49f7aff3df049b19d150ecb11e4b171: Generated new TSK 0
I20260925 14:57:24.715314 6808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:24.726696 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6725
W20260925 14:57:24.727814 6788 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.57:60831
W20260925 14:57:24.728645 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:24.728816 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 6725
2026-09-25T14:57:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:24 dist-test-slave-v5vl krb5kdc[6824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:24 dist-test-slave-v5vl krb5kdc[6824](info): set up 2 sockets
Sep 25 14:57:24 dist-test-slave-v5vl krb5kdc[6824](info): commencing operation
krb5kdc: starting...
W20260925 14:57:25.825129 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:25 dist-test-slave-v5vl krb5kdc[6824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:26.073575 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:26.403834 6837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:26.404153 6837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:26.404331 6837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:26.412052 6837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:26.412192 6837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:26.412297 6837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:26.412395 6837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:26.412467 6837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:26.422901 6837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:26.424279 6837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:26.425694 6837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:26.437601 6843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:26.437634 6842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:26.437665 6845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:26.438377 6837 server_base.cc:1034] running on GCE node
I20260925 14:57:26.439349 6837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:26.440884 6837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:26.442142 6837 hybrid_clock.cc:648] HybridClock initialized: now 1790348246442088 us; error 42 us; skew 500 ppm
I20260925 14:57:26.442533 6837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:26 dist-test-slave-v5vl krb5kdc[6824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:26.448701 6837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:26.450222 6837 webserver.cc:460] Webserver started at http://127.0.0.1:34643/ using document root <none> and password file <none>
I20260925 14:57:26.450919 6837 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:26.451059 6837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:26.451359 6837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:26.454516 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "dbaa95569bd94e54aa0b140f15086548"
format_stamp: "Formatted at 2026-09-25 14:57:26 on dist-test-slave-v5vl"
server_key: "957242aeb9c19a3345595f9e4e91574d"
server_key_iv: "3f2127dd7b559980d0ec7caf838d5994"
server_key_version: "encryptionkey@0"
I20260925 14:57:26.455341 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "dbaa95569bd94e54aa0b140f15086548"
format_stamp: "Formatted at 2026-09-25 14:57:26 on dist-test-slave-v5vl"
server_key: "957242aeb9c19a3345595f9e4e91574d"
server_key_iv: "3f2127dd7b559980d0ec7caf838d5994"
server_key_version: "encryptionkey@0"
I20260925 14:57:26.460881 6837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 14:57:26.464324 6852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:26.465675 6837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:26.465819 6837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "dbaa95569bd94e54aa0b140f15086548"
format_stamp: "Formatted at 2026-09-25 14:57:26 on dist-test-slave-v5vl"
server_key: "957242aeb9c19a3345595f9e4e91574d"
server_key_iv: "3f2127dd7b559980d0ec7caf838d5994"
server_key_version: "encryptionkey@0"
I20260925 14:57:26.465982 6837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:26.477483 6837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:26.481508 6837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:26.481743 6837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:26.499608 6837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45615
I20260925 14:57:26.499655 6904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45615 every 8 connection(s)
I20260925 14:57:26.501066 6837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:26.504891 6905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:26.508152 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 6837
I20260925 14:57:26.508353 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:26.508898 17943 external_mini_cluster.cc:1428] Setting key bf58688493ebb0196f7375b464bb7d67
I20260925 14:57:26.516819 6905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548: Bootstrap starting.
I20260925 14:57:26.519757 6905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:26.520793 6905 log.cc:824] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548: Log is configured to *not* fsync() on all Append() calls
Sep 25 14:57:26 dist-test-slave-v5vl krb5kdc[6824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:57:26.523999 6905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548: No bootstrap required, opened a new log
I20260925 14:57:26.531071 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:26.510794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:36809 (local address 127.0.0.1:45615)
0925 14:57:26.511812 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:26.511824 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:26.511861 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:57:26.513504 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:26.513528 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:26.513924 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:26.514333 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:26.515705 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:26.517052 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:26.518227 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:26.518895 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:26.519072 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:26.523752 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:26.523759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:26.523771 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:26.523803 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:26.526841 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:26.527743 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:26.527748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:26.527755 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:26.527918 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:26.528364 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:26.528367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:26.528369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:26.528950 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:26.529178 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:26.529715 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 14:57:26.530017 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":559,"threads_started":1}
I20260925 14:57:26.533018 6905 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbaa95569bd94e54aa0b140f15086548" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45615 } }
I20260925 14:57:26.533463 6905 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:26.533618 6905 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbaa95569bd94e54aa0b140f15086548, State: Initialized, Role: FOLLOWER
I20260925 14:57:26.534235 6905 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbaa95569bd94e54aa0b140f15086548" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45615 } }
I20260925 14:57:26.534582 6905 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:26.534725 6905 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:26.534910 6905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:26.535301 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:26.537211 6905 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbaa95569bd94e54aa0b140f15086548" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45615 } }
I20260925 14:57:26.537679 6905 leader_election.cc:302] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbaa95569bd94e54aa0b140f15086548; no voters:
I20260925 14:57:26.538332 6905 leader_election.cc:288] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:26.538426 6912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:26.539603 6912 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [term 1 LEADER]: Becoming Leader. State: Replica: dbaa95569bd94e54aa0b140f15086548, State: Running, Role: LEADER
I20260925 14:57:26.540731 6905 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:26.540684 6912 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbaa95569bd94e54aa0b140f15086548" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45615 } }
I20260925 14:57:26.550334 6915 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbaa95569bd94e54aa0b140f15086548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbaa95569bd94e54aa0b140f15086548" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45615 } } }
I20260925 14:57:26.550734 6915 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:26.550304 6916 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbaa95569bd94e54aa0b140f15086548. Latest consensus state: current_term: 1 leader_uuid: "dbaa95569bd94e54aa0b140f15086548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbaa95569bd94e54aa0b140f15086548" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45615 } } }
I20260925 14:57:26.551347 6916 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:26.556025 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:26.540133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:58369 (local address 127.0.0.1:45615)
0925 14:57:26.540301 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:26.540306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:26.540649 (+ 343us) server_negotiation.cc:407] Connection header received
0925 14:57:26.540864 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:26.540869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:26.540969 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:26.541062 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:26.542486 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:26.543313 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:26.544501 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:26.544983 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:26.545105 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:26.547716 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:26.547720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:26.547723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:26.547751 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:26.551673 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:26.552870 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:26.552878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:26.552884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:26.553056 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:26.553710 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:26.553718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:26.553723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:26.554191 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:26.554468 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:26.555290 (+ 822us) server_negotiation.cc:299] Negotiation successful
0925 14:57:26.555523 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:57:26.561255 6924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:26.566293 6924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:26.576395 6924 catalog_manager.cc:1347] Generated new cluster ID: 71041f92203e449783969e0a1a76b3f9
I20260925 14:57:26.576968 6924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:26.586762 6924 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:26.588414 6924 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:26.599445 6924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbaa95569bd94e54aa0b140f15086548: Generated new TSK 0
I20260925 14:57:26.600641 6924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:26.613183 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6837
I20260925 14:57:26.614385 6899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:26.704213 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 6837
W20260925 14:57:26.721482 6822 connection.cc:535] client connection to 127.0.0.1:45615 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45615): Connection reset by peer (error 104)
2026-09-25T14:57:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:26 dist-test-slave-v5vl krb5kdc[6937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:26 dist-test-slave-v5vl krb5kdc[6937](info): set up 2 sockets
Sep 25 14:57:26 dist-test-slave-v5vl krb5kdc[6937](info): commencing operation
krb5kdc: starting...
W20260925 14:57:27.785097 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.011s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:27 dist-test-slave-v5vl krb5kdc[6937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:28.038057 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:28.373073 6950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:28.373407 6950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:28.373558 6950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:28.380715 6950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:28.380856 6950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:28.380957 6950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:28.381068 6950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:28.381155 6950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:28.391139 6950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:28.392395 6950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:28.393740 6950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:28.405417 6955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:28.405455 6956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:28.405491 6958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:28.406284 6950 server_base.cc:1034] running on GCE node
I20260925 14:57:28.407085 6950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:28.408476 6950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:28.409688 6950 hybrid_clock.cc:648] HybridClock initialized: now 1790348248409648 us; error 34 us; skew 500 ppm
I20260925 14:57:28.410025 6950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:28 dist-test-slave-v5vl krb5kdc[6937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:28.413017 6950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:28.414449 6950 webserver.cc:460] Webserver started at http://127.0.0.1:33767/ using document root <none> and password file <none>
I20260925 14:57:28.415127 6950 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:28.415268 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:28.415561 6950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:28.418605 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "974942d63c1140d783a33957588b32b5"
format_stamp: "Formatted at 2026-09-25 14:57:28 on dist-test-slave-v5vl"
server_key: "0ec21bc52f68062a1f8d060274eeff9d"
server_key_iv: "76571e25a3fef50fbf166d78f7136c9f"
server_key_version: "encryptionkey@0"
I20260925 14:57:28.419358 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "974942d63c1140d783a33957588b32b5"
format_stamp: "Formatted at 2026-09-25 14:57:28 on dist-test-slave-v5vl"
server_key: "0ec21bc52f68062a1f8d060274eeff9d"
server_key_iv: "76571e25a3fef50fbf166d78f7136c9f"
server_key_version: "encryptionkey@0"
I20260925 14:57:28.424566 6950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 14:57:28.427958 6965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:28.429334 6950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:57:28.429513 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "974942d63c1140d783a33957588b32b5"
format_stamp: "Formatted at 2026-09-25 14:57:28 on dist-test-slave-v5vl"
server_key: "0ec21bc52f68062a1f8d060274eeff9d"
server_key_iv: "76571e25a3fef50fbf166d78f7136c9f"
server_key_version: "encryptionkey@0"
I20260925 14:57:28.429708 6950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:28.447482 6950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:28.451433 6950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:28.451763 6950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:28.470125 6950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33385
I20260925 14:57:28.470149 7016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33385 every 8 connection(s)
I20260925 14:57:28.471505 6950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:28.474992 7017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:28.475966 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 6950
I20260925 14:57:28.476251 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:28.476752 17943 external_mini_cluster.cc:1428] Setting key 24e831ef05422c0035a72c285ec4d5b7
I20260925 14:57:28.488490 7017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5: Bootstrap starting.
I20260925 14:57:28.490046 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.478630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:57559 (local address 127.0.0.1:33385)
0925 14:57:28.479405 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.479416 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.479456 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:57:28.481115 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.481137 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.481549 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.481886 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:28.482456 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.483704 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.485665 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.486375 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.486507 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.487759 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:28.487765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:28.487779 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:28.488298 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:28.488374 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:28.488399 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:28.488529 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.488905 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.489095 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":349,"threads_started":1}
I20260925 14:57:28.491698 7017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:28.492867 7017 log.cc:824] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:28.493741 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:28.495558 7017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5: No bootstrap required, opened a new log
I20260925 14:57:28.504400 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.497992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:43113 (local address 127.0.0.1:33385)
0925 14:57:28.498174 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.498180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.498640 (+ 460us) server_negotiation.cc:407] Connection header received
0925 14:57:28.498778 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.498785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.498858 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.498978 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:28.499433 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.500639 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.501839 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.502557 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.502686 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.503668 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:28.503674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:28.503679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:28.503698 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:28.503756 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:28.503774 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:28.503893 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.504034 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.504081 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:57:28.504431 7017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974942d63c1140d783a33957588b32b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } }
I20260925 14:57:28.504907 7017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:28.505034 7017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974942d63c1140d783a33957588b32b5, State: Initialized, Role: FOLLOWER
I20260925 14:57:28.505604 7017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974942d63c1140d783a33957588b32b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } }
I20260925 14:57:28.505831 7017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:28.505964 7017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:28.506129 7017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:28.508212 7017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974942d63c1140d783a33957588b32b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } }
I20260925 14:57:28.508723 7017 leader_election.cc:302] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 974942d63c1140d783a33957588b32b5; no voters:
I20260925 14:57:28.509264 7017 leader_election.cc:288] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:28.509359 7027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:28.510394 7027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [term 1 LEADER]: Becoming Leader. State: Replica: 974942d63c1140d783a33957588b32b5, State: Running, Role: LEADER
I20260925 14:57:28.510896 7027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974942d63c1140d783a33957588b32b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } }
I20260925 14:57:28.511582 7017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:28.513863 7028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "974942d63c1140d783a33957588b32b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974942d63c1140d783a33957588b32b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } } }
I20260925 14:57:28.514317 7028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:28.515079 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 974942d63c1140d783a33957588b32b5. Latest consensus state: current_term: 1 leader_uuid: "974942d63c1140d783a33957588b32b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974942d63c1140d783a33957588b32b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } } }
I20260925 14:57:28.515630 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:28.521692 7036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:28.526631 7036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:28.536196 7036 catalog_manager.cc:1347] Generated new cluster ID: d48412435d934559907699810d576330
I20260925 14:57:28.536314 7036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:28.555035 7036 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:28.556407 7036 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:28.567624 7036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 974942d63c1140d783a33957588b32b5: Generated new TSK 0
I20260925 14:57:28.568689 7036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:28.576182 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6950
W20260925 14:57:28.577701 7012 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.57:57559
W20260925 14:57:28.578420 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 6950: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:28.578555 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 6950
W20260925 14:57:28.595624 6935 connection.cc:535] client connection to 127.0.0.1:33385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33385): Connection reset by peer (error 104)
2026-09-25T14:57:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:28 dist-test-slave-v5vl krb5kdc[7049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:28 dist-test-slave-v5vl krb5kdc[7049](info): set up 2 sockets
Sep 25 14:57:28 dist-test-slave-v5vl krb5kdc[7049](info): commencing operation
krb5kdc: starting...
W20260925 14:57:29.665185 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:29 dist-test-slave-v5vl krb5kdc[7049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:29.911751 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:30.239559 7062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:30.239841 7062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:30.239939 7062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:30.247057 7062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:30.247181 7062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:30.247231 7062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:30.247292 7062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:30.247335 7062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:30.257622 7062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:30.258986 7062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:30.260245 7062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:30.271724 7068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:30.271685 7070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:30.271936 7067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:30.272742 7062 server_base.cc:1034] running on GCE node
I20260925 14:57:30.273485 7062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:30.274914 7062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:30.276139 7062 hybrid_clock.cc:648] HybridClock initialized: now 1790348250276099 us; error 50 us; skew 500 ppm
I20260925 14:57:30.276484 7062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:30 dist-test-slave-v5vl krb5kdc[7049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:30.279625 7062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:30.281082 7062 webserver.cc:460] Webserver started at http://127.0.0.1:38833/ using document root <none> and password file <none>
I20260925 14:57:30.281785 7062 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:30.281930 7062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:30.282222 7062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:30.285300 7062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-v5vl"
server_key: "1951a430c83f1e74c3f1ac57fb3a9f11"
server_key_iv: "4b523149d3869075b76907d011988574"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.286059 7062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-v5vl"
server_key: "1951a430c83f1e74c3f1ac57fb3a9f11"
server_key_iv: "4b523149d3869075b76907d011988574"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.291126 7062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:57:30.294431 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.295688 7062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:30.295864 7062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-v5vl"
server_key: "1951a430c83f1e74c3f1ac57fb3a9f11"
server_key_iv: "4b523149d3869075b76907d011988574"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.296078 7062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:30.301576 7062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:30.305837 7062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:30.306129 7062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:30.323657 7062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37983
I20260925 14:57:30.323702 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37983 every 8 connection(s)
I20260925 14:57:30.325120 7062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:30.325392 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7062
I20260925 14:57:30.325572 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:30.326195 17943 external_mini_cluster.cc:1428] Setting key 337b8e1ae215345ee9db867dd110b53b
I20260925 14:57:30.329830 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:30.334674 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.328638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:53395 (local address 127.0.0.1:37983)
0925 14:57:30.329358 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.329378 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.329417 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:57:30.331092 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.331113 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.331434 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.331804 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:30.333107 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:30.333113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:30.333128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:30.333139 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:30.333202 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:30.333222 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:30.333465 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.333599 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.333780 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":299,"threads_started":1}
I20260925 14:57:30.338454 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:30.340849 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee: Bootstrap starting.
I20260925 14:57:30.344111 7129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:30.345048 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.342580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.57:54897 (local address 127.0.0.1:37983)
0925 14:57:30.342759 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.342767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.343122 (+ 355us) server_negotiation.cc:407] Connection header received
0925 14:57:30.343311 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.343315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.343366 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.343459 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:30.344397 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:30.344402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:30.344407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:30.344430 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:30.344470 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:30.344487 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:30.344638 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.344714 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.344772 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:57:30.345376 7129 log.cc:824] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:30.348201 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee: No bootstrap required, opened a new log
I20260925 14:57:30.355976 7129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } }
I20260925 14:57:30.356334 7129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:30.356405 7129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3c4f2ae56040e0ad4b1fbd624eb2ee, State: Initialized, Role: FOLLOWER
I20260925 14:57:30.356997 7129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } }
I20260925 14:57:30.357191 7129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:30.357290 7129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:30.357421 7129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:30.359588 7129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } }
I20260925 14:57:30.360072 7129 leader_election.cc:302] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c3c4f2ae56040e0ad4b1fbd624eb2ee; no voters:
I20260925 14:57:30.360797 7129 leader_election.cc:288] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:30.360909 7139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:30.361618 7139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [term 1 LEADER]: Becoming Leader. State: Replica: 9c3c4f2ae56040e0ad4b1fbd624eb2ee, State: Running, Role: LEADER
I20260925 14:57:30.362191 7139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } }
I20260925 14:57:30.362860 7129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:30.365090 7140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } } }
I20260925 14:57:30.365638 7140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:30.366345 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c3c4f2ae56040e0ad4b1fbd624eb2ee. Latest consensus state: current_term: 1 leader_uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3c4f2ae56040e0ad4b1fbd624eb2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } } }
I20260925 14:57:30.366704 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:30.367017 7148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:30.372191 7148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:30.381158 7148 catalog_manager.cc:1347] Generated new cluster ID: 512344e3150d41968c76627f8a77f2f7
I20260925 14:57:30.381279 7148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:30.412503 7148 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:30.412985 7148 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:30.419593 7148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c3c4f2ae56040e0ad4b1fbd624eb2ee: Generated new TSK 0
I20260925 14:57:30.420519 7148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:30.426924 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7062
W20260925 14:57:30.428529 7124 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.57:53395
W20260925 14:57:30.429319 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7062: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:30.429461 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7062
2026-09-25T14:57:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:30 dist-test-slave-v5vl krb5kdc[7160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:30 dist-test-slave-v5vl krb5kdc[7160](info): set up 2 sockets
Sep 25 14:57:30 dist-test-slave-v5vl krb5kdc[7160](info): commencing operation
krb5kdc: starting...
W20260925 14:57:31.517151 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:31 dist-test-slave-v5vl krb5kdc[7160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:57:31.767477 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:36847
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:32.103072 7173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:32.103404 7173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:32.103569 7173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:32.110699 7173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:32.110849 7173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:32.110944 7173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:32.111054 7173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:32.111131 7173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:32.120985 7173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:36847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:36847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:32.122305 7173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:32.123570 7173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:32.135787 7181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:32.135825 7178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:32.135735 7179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:32.136720 7173 server_base.cc:1034] running on GCE node
I20260925 14:57:32.137462 7173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:32.138892 7173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:32.140110 7173 hybrid_clock.cc:648] HybridClock initialized: now 1790348252140062 us; error 47 us; skew 500 ppm
I20260925 14:57:32.140465 7173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:32 dist-test-slave-v5vl krb5kdc[7160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348252, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:32.143469 7173 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:57:32.144927 7173 webserver.cc:460] Webserver started at http://127.17.133.254:44353/ using document root <none> and password file <none>
I20260925 14:57:32.145601 7173 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:32.145738 7173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:32.146027 7173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:32.148993 7173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "1ad95493aa154d3cbf10719cee87fd52"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-v5vl"
server_key: "d192eaa1d95723dab87bcfb234e86c4c"
server_key_iv: "6c13e63ce6356ff0c17bbb5aa0b2a525"
server_key_version: "encryptionkey@0"
I20260925 14:57:32.149752 7173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "1ad95493aa154d3cbf10719cee87fd52"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-v5vl"
server_key: "d192eaa1d95723dab87bcfb234e86c4c"
server_key_iv: "6c13e63ce6356ff0c17bbb5aa0b2a525"
server_key_version: "encryptionkey@0"
I20260925 14:57:32.154819 7173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:57:32.158319 7188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:32.159629 7173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:57:32.159809 7173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "1ad95493aa154d3cbf10719cee87fd52"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-v5vl"
server_key: "d192eaa1d95723dab87bcfb234e86c4c"
server_key_iv: "6c13e63ce6356ff0c17bbb5aa0b2a525"
server_key_version: "encryptionkey@0"
I20260925 14:57:32.160003 7173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:32.174896 7173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:32.180267 7173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:32.180577 7173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:32.198945 7173 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:36847
I20260925 14:57:32.198942 7239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:36847 every 8 connection(s)
I20260925 14:57:32.200367 7173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:32.204121 7240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:32.209106 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7173
I20260925 14:57:32.209264 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:32.209690 17943 external_mini_cluster.cc:1428] Setting key fbb8c08bf37d09f09251e5981ec24666
I20260925 14:57:32.216136 7240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52: Bootstrap starting.
I20260925 14:57:32.219858 7240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:32.221074 7240 log.cc:824] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:32.223948 7240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52: No bootstrap required, opened a new log
Sep 25 14:57:32 dist-test-slave-v5vl krb5kdc[7160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:32.233217 7240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad95493aa154d3cbf10719cee87fd52" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 36847 } }
I20260925 14:57:32.233620 7240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:32.233731 7240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad95493aa154d3cbf10719cee87fd52, State: Initialized, Role: FOLLOWER
I20260925 14:57:32.233891 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:32.211431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46349 (local address 127.17.133.254:36847)
0925 14:57:32.212262 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:32.212277 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:32.212335 (+ 58us) server_negotiation.cc:407] Connection header received
0925 14:57:32.214796 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:32.214824 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:32.215329 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:32.215753 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:32.217121 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.219180 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.220434 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.221540 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.221779 (+ 239us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:32.226050 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:32.226061 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:32.226081 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:32.226129 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:32.229590 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.230592 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.230598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.230604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.230753 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.231206 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.231210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.231212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.231724 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:32.231966 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:32.232425 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 14:57:32.232777 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":316,"threads_started":1}
I20260925 14:57:32.234465 7240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad95493aa154d3cbf10719cee87fd52" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 36847 } }
I20260925 14:57:32.234723 7240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:32.234887 7240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:32.235071 7240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:32.237574 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:32.237694 7240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad95493aa154d3cbf10719cee87fd52" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 36847 } }
I20260925 14:57:32.238178 7240 leader_election.cc:302] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ad95493aa154d3cbf10719cee87fd52; no voters:
I20260925 14:57:32.238751 7240 leader_election.cc:288] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:32.238886 7245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:32.239501 7245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [term 1 LEADER]: Becoming Leader. State: Replica: 1ad95493aa154d3cbf10719cee87fd52, State: Running, Role: LEADER
I20260925 14:57:32.240113 7245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad95493aa154d3cbf10719cee87fd52" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 36847 } }
I20260925 14:57:32.243119 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ad95493aa154d3cbf10719cee87fd52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad95493aa154d3cbf10719cee87fd52" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 36847 } } }
I20260925 14:57:32.243660 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:32.245419 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ad95493aa154d3cbf10719cee87fd52. Latest consensus state: current_term: 1 leader_uuid: "1ad95493aa154d3cbf10719cee87fd52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad95493aa154d3cbf10719cee87fd52" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 36847 } } }
I20260925 14:57:32.246065 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:32.254459 7240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:32.254652 7251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:32.261783 7251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:32.274833 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:32.255631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49759 (local address 127.17.133.254:36847)
0925 14:57:32.255865 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:32.255875 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:32.257427 (+ 1552us) server_negotiation.cc:407] Connection header received
0925 14:57:32.257742 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:32.257749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:32.257847 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:32.257936 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:32.260064 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.261267 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.263166 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.263863 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.263983 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:32.267333 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:32.267340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:32.267345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:32.267384 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:32.271036 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.271961 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.271966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.271969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.272130 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.272739 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.272745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.272748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.273161 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:32.273412 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:32.274117 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 14:57:32.274329 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:57:32.276571 7251 catalog_manager.cc:1347] Generated new cluster ID: bf2ee884b7354a988c9649e67076135a
I20260925 14:57:32.276700 7251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:32.287648 7251 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:32.289371 7251 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:32.308239 7251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ad95493aa154d3cbf10719cee87fd52: Generated new TSK 0
I20260925 14:57:32.309484 7251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:32.317205 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7173
I20260925 14:57:32.318423 7235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:32.407635 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7173
W20260925 14:57:32.424135 7158 connection.cc:535] client connection to 127.17.133.254:36847 recv error: Network error: failed to read from TLS socket (remote: 127.17.133.254:36847): Connection reset by peer (error 104)
2026-09-25T14:57:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:32 dist-test-slave-v5vl krb5kdc[7273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:32 dist-test-slave-v5vl krb5kdc[7273](info): set up 2 sockets
Sep 25 14:57:32 dist-test-slave-v5vl krb5kdc[7273](info): commencing operation
krb5kdc: starting...
W20260925 14:57:33.509239 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:33 dist-test-slave-v5vl krb5kdc[7273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:57:33.755967 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:32905
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:32905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:34.081964 7286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:34.082273 7286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:34.082365 7286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:34.089610 7286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:34.089736 7286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:34.089784 7286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:34.089859 7286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:34.089903 7286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:34.100407 7286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:32905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:32905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:34.101708 7286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:34.102972 7286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:34.114596 7292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:34.114661 7291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:34.114681 7294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:34.115576 7286 server_base.cc:1034] running on GCE node
I20260925 14:57:34.116292 7286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:34.117676 7286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:34.118885 7286 hybrid_clock.cc:648] HybridClock initialized: now 1790348254118838 us; error 36 us; skew 500 ppm
I20260925 14:57:34.119232 7286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:34 dist-test-slave-v5vl krb5kdc[7273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:34.122228 7286 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:57:34.123662 7286 webserver.cc:460] Webserver started at http://127.17.133.254:45467/ using document root <none> and password file <none>
I20260925 14:57:34.124341 7286 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:34.124466 7286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:34.124812 7286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:34.127708 7286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "ca9888fdac3749ab84380b1d1ee07607"
format_stamp: "Formatted at 2026-09-25 14:57:34 on dist-test-slave-v5vl"
server_key: "0219442ffa4a9035bc8cbd76e883e706"
server_key_iv: "37bf4f17aa9ffda7d299aaf565d8a909"
server_key_version: "encryptionkey@0"
I20260925 14:57:34.128412 7286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "ca9888fdac3749ab84380b1d1ee07607"
format_stamp: "Formatted at 2026-09-25 14:57:34 on dist-test-slave-v5vl"
server_key: "0219442ffa4a9035bc8cbd76e883e706"
server_key_iv: "37bf4f17aa9ffda7d299aaf565d8a909"
server_key_version: "encryptionkey@0"
I20260925 14:57:34.133421 7286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 14:57:34.136875 7301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:34.138140 7286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:57:34.138338 7286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "ca9888fdac3749ab84380b1d1ee07607"
format_stamp: "Formatted at 2026-09-25 14:57:34 on dist-test-slave-v5vl"
server_key: "0219442ffa4a9035bc8cbd76e883e706"
server_key_iv: "37bf4f17aa9ffda7d299aaf565d8a909"
server_key_version: "encryptionkey@0"
I20260925 14:57:34.138527 7286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:34.152995 7286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:34.157159 7286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:34.157449 7286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:34.177975 7286 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:32905
I20260925 14:57:34.177999 7352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:32905 every 8 connection(s)
I20260925 14:57:34.179781 7286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:34.183627 7353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:34.189881 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7286
I20260925 14:57:34.190062 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:34.190629 17943 external_mini_cluster.cc:1428] Setting key 28336e05d060ba1f96a6975cc2a9cd2c
I20260925 14:57:34.194725 7353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607: Bootstrap starting.
I20260925 14:57:34.198160 7353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:34.199494 7353 log.cc:824] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:34.202857 7353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607: No bootstrap required, opened a new log
I20260925 14:57:34.204519 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:34.192448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54589 (local address 127.17.133.254:32905)
0925 14:57:34.193284 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:34.193295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:34.193335 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:57:34.195037 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:34.195059 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:34.195468 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:34.195784 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:34.196369 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.197657 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.199661 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.200818 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.201018 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:34.202270 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:34.202278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:34.202292 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:34.202809 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:34.202883 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:34.202909 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:34.203080 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:34.203383 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 14:57:34.203566 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":438,"threads_started":1}
I20260925 14:57:34.208410 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:34.211300 7353 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9888fdac3749ab84380b1d1ee07607" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 32905 } }
I20260925 14:57:34.211665 7353 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:34.211776 7353 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9888fdac3749ab84380b1d1ee07607, State: Initialized, Role: FOLLOWER
I20260925 14:57:34.212342 7353 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9888fdac3749ab84380b1d1ee07607" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 32905 } }
I20260925 14:57:34.212678 7353 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:34.212823 7353 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:34.213081 7353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:34.216051 7353 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9888fdac3749ab84380b1d1ee07607" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 32905 } }
I20260925 14:57:34.216605 7353 leader_election.cc:302] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca9888fdac3749ab84380b1d1ee07607; no voters:
I20260925 14:57:34.217269 7353 leader_election.cc:288] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:34.217406 7363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:34.218987 7363 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [term 1 LEADER]: Becoming Leader. State: Replica: ca9888fdac3749ab84380b1d1ee07607, State: Running, Role: LEADER
I20260925 14:57:34.219384 7353 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:34.219625 7363 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9888fdac3749ab84380b1d1ee07607" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 32905 } }
I20260925 14:57:34.220335 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:34.212724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34349 (local address 127.17.133.254:32905)
0925 14:57:34.212928 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:34.212936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:34.213386 (+ 450us) server_negotiation.cc:407] Connection header received
0925 14:57:34.213569 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:34.213572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:34.213628 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:34.213711 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:34.214280 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.215006 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.216786 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.217463 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.217646 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:34.219428 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:34.219434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:34.219438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:34.219459 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:34.219530 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:34.219562 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:34.219690 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:34.219963 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 14:57:34.220028 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:57:34.222786 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9888fdac3749ab84380b1d1ee07607. Latest consensus state: current_term: 1 leader_uuid: "ca9888fdac3749ab84380b1d1ee07607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9888fdac3749ab84380b1d1ee07607" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 32905 } } }
I20260925 14:57:34.222750 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca9888fdac3749ab84380b1d1ee07607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9888fdac3749ab84380b1d1ee07607" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 32905 } } }
I20260925 14:57:34.223253 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:34.223248 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:34.229286 7369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:34.234659 7369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:34.244549 7369 catalog_manager.cc:1347] Generated new cluster ID: f59680c85f144baf95dc1600c8599189
I20260925 14:57:34.244720 7369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:34.253772 7369 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:34.255028 7369 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:34.262955 7369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca9888fdac3749ab84380b1d1ee07607: Generated new TSK 0
I20260925 14:57:34.263887 7369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:34.278862 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7286
W20260925 14:57:34.279989 7348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54589
W20260925 14:57:34.280685 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7286: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:34.280823 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7286
W20260925 14:57:34.297178 7271 connection.cc:535] client connection to 127.17.133.254:32905 recv error: Network error: failed to read from TLS socket (remote: 127.17.133.254:32905): Connection reset by peer (error 104)
2026-09-25T14:57:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:34 dist-test-slave-v5vl krb5kdc[7384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:34 dist-test-slave-v5vl krb5kdc[7384](info): set up 2 sockets
Sep 25 14:57:34 dist-test-slave-v5vl krb5kdc[7384](info): commencing operation
krb5kdc: starting...
W20260925 14:57:35.385108 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:35 dist-test-slave-v5vl krb5kdc[7384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:57:35.639058 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46451
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:46451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:35.971241 7397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:35.971534 7397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:35.971629 7397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:35.978735 7397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:35.978844 7397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:35.978891 7397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:35.978951 7397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:35.978995 7397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:35.988740 7397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:46451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:35.990003 7397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:35.991268 7397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:36.003281 7405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:36.003297 7402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:36.003360 7403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:36.004060 7397 server_base.cc:1034] running on GCE node
I20260925 14:57:36.004982 7397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:36.006479 7397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:36.007746 7397 hybrid_clock.cc:648] HybridClock initialized: now 1790348256007689 us; error 59 us; skew 500 ppm
I20260925 14:57:36.008123 7397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:36 dist-test-slave-v5vl krb5kdc[7384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348256, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:36.011268 7397 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:57:36.012809 7397 webserver.cc:460] Webserver started at http://127.17.133.254:34595/ using document root <none> and password file <none>
I20260925 14:57:36.013486 7397 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:36.013634 7397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:36.013945 7397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:36.017009 7397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "2dfbca9b01174257873dc494e4c4999d"
format_stamp: "Formatted at 2026-09-25 14:57:36 on dist-test-slave-v5vl"
server_key: "8b314b9bbf2e7414f5ec83e28d87c663"
server_key_iv: "c821d04b9d5de426d5bbc93aa8f54e42"
server_key_version: "encryptionkey@0"
I20260925 14:57:36.017827 7397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "2dfbca9b01174257873dc494e4c4999d"
format_stamp: "Formatted at 2026-09-25 14:57:36 on dist-test-slave-v5vl"
server_key: "8b314b9bbf2e7414f5ec83e28d87c663"
server_key_iv: "c821d04b9d5de426d5bbc93aa8f54e42"
server_key_version: "encryptionkey@0"
I20260925 14:57:36.022917 7397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:57:36.026460 7412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:36.027895 7397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:57:36.028090 7397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "2dfbca9b01174257873dc494e4c4999d"
format_stamp: "Formatted at 2026-09-25 14:57:36 on dist-test-slave-v5vl"
server_key: "8b314b9bbf2e7414f5ec83e28d87c663"
server_key_iv: "c821d04b9d5de426d5bbc93aa8f54e42"
server_key_version: "encryptionkey@0"
I20260925 14:57:36.028308 7397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:36.034274 7397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:36.038389 7397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:36.038697 7397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:36.057507 7397 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:46451
I20260925 14:57:36.057556 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:46451 every 8 connection(s)
I20260925 14:57:36.058991 7397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:36.062736 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:36.063836 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7397
I20260925 14:57:36.064014 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:36.064558 17943 external_mini_cluster.cc:1428] Setting key a11b61b195045e3edfc6a9c8a7adec49
I20260925 14:57:36.072259 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:36.066320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58465 (local address 127.17.133.254:46451)
0925 14:57:36.067121 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:36.067131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:36.067178 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:57:36.068917 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:36.068943 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:36.069298 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:36.069639 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:36.070686 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:36.070693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:36.070706 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:36.070720 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:36.070792 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:36.070812 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:36.070976 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:36.071103 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:57:36.071323 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":382,"threads_started":1}
I20260925 14:57:36.075850 7464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d: Bootstrap starting.
I20260925 14:57:36.076009 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:36.079031 7464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:36.080212 7464 log.cc:824] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:36.082921 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:36.080342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50935 (local address 127.17.133.254:46451)
0925 14:57:36.080554 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:36.080561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:36.080985 (+ 424us) server_negotiation.cc:407] Connection header received
0925 14:57:36.081167 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:36.081170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:36.081238 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:36.081324 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:36.082303 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:36.082308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:36.082312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:36.082330 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:36.082375 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:36.082394 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:36.082504 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:36.082587 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 14:57:36.082642 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:57:36.083503 7464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d: No bootstrap required, opened a new log
I20260925 14:57:36.090516 7464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfbca9b01174257873dc494e4c4999d" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46451 } }
I20260925 14:57:36.090875 7464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:36.090993 7464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfbca9b01174257873dc494e4c4999d, State: Initialized, Role: FOLLOWER
I20260925 14:57:36.091555 7464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfbca9b01174257873dc494e4c4999d" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46451 } }
I20260925 14:57:36.091796 7464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:36.091950 7464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:36.092120 7464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:36.094312 7464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfbca9b01174257873dc494e4c4999d" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46451 } }
I20260925 14:57:36.094821 7464 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfbca9b01174257873dc494e4c4999d; no voters:
I20260925 14:57:36.095386 7464 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:36.095513 7474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:36.096191 7474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [term 1 LEADER]: Becoming Leader. State: Replica: 2dfbca9b01174257873dc494e4c4999d, State: Running, Role: LEADER
I20260925 14:57:36.096781 7474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfbca9b01174257873dc494e4c4999d" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46451 } }
I20260925 14:57:36.097433 7464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:36.099582 7476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfbca9b01174257873dc494e4c4999d. Latest consensus state: current_term: 1 leader_uuid: "2dfbca9b01174257873dc494e4c4999d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfbca9b01174257873dc494e4c4999d" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46451 } } }
I20260925 14:57:36.100040 7476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:36.101140 7475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dfbca9b01174257873dc494e4c4999d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfbca9b01174257873dc494e4c4999d" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46451 } } }
I20260925 14:57:36.101574 7475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:36.105262 7481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:36.110826 7481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:36.120172 7481 catalog_manager.cc:1347] Generated new cluster ID: 877fd37a8a7c4ee890957e343aa6eb88
I20260925 14:57:36.120327 7481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:36.138641 7481 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:36.139194 7481 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:36.154207 7481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dfbca9b01174257873dc494e4c4999d: Generated new TSK 0
I20260925 14:57:36.155122 7481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:36.163889 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7397
W20260925 14:57:36.165483 7459 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58465
W20260925 14:57:36.166180 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7397: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:36.166339 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7397
2026-09-25T14:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:36 dist-test-slave-v5vl krb5kdc[7495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:36 dist-test-slave-v5vl krb5kdc[7495](info): set up 2 sockets
Sep 25 14:57:36 dist-test-slave-v5vl krb5kdc[7495](info): commencing operation
krb5kdc: starting...
W20260925 14:57:37.257115 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.012s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:37 dist-test-slave-v5vl krb5kdc[7495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:57:37.505898 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:39909
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:39909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:37.836758 7508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:37.837021 7508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:37.837201 7508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:37.844862 7508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:37.845033 7508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:37.845170 7508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:37.845276 7508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:37.845356 7508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:37.855257 7508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:39909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:39909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:37.856575 7508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:37.857895 7508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:37.869243 7514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:37.869310 7516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:37.869268 7513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:37.870445 7508 server_base.cc:1034] running on GCE node
I20260925 14:57:37.871222 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:37.872551 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:37.873827 7508 hybrid_clock.cc:648] HybridClock initialized: now 1790348257873785 us; error 37 us; skew 500 ppm
I20260925 14:57:37.874194 7508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:37 dist-test-slave-v5vl krb5kdc[7495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348257, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:37.891415 7508 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:57:37.893046 7508 webserver.cc:460] Webserver started at http://127.17.133.254:36111/ using document root <none> and password file <none>
I20260925 14:57:37.893740 7508 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:37.893846 7508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:37.894090 7508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:37.897048 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "5fd009d274cb42c29470f2db78702db3"
format_stamp: "Formatted at 2026-09-25 14:57:37 on dist-test-slave-v5vl"
server_key: "30ef4b71b0bf9dc49bcf40357cd92d33"
server_key_iv: "f0446c45354dca9ad362f446d82b286e"
server_key_version: "encryptionkey@0"
I20260925 14:57:37.897737 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "5fd009d274cb42c29470f2db78702db3"
format_stamp: "Formatted at 2026-09-25 14:57:37 on dist-test-slave-v5vl"
server_key: "30ef4b71b0bf9dc49bcf40357cd92d33"
server_key_iv: "f0446c45354dca9ad362f446d82b286e"
server_key_version: "encryptionkey@0"
I20260925 14:57:37.902771 7508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:57:37.906100 7523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:37.907416 7508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 14:57:37.907552 7508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "5fd009d274cb42c29470f2db78702db3"
format_stamp: "Formatted at 2026-09-25 14:57:37 on dist-test-slave-v5vl"
server_key: "30ef4b71b0bf9dc49bcf40357cd92d33"
server_key_iv: "f0446c45354dca9ad362f446d82b286e"
server_key_version: "encryptionkey@0"
I20260925 14:57:37.907704 7508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:37.925153 7508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:37.929080 7508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:37.929389 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:37.947757 7508 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:39909
I20260925 14:57:37.947774 7574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:39909 every 8 connection(s)
I20260925 14:57:37.949093 7508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:37.950045 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7508
I20260925 14:57:37.950237 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:37.950636 17943 external_mini_cluster.cc:1428] Setting key 1ac5615b9a95b7eeb1e56a1f56f30719
I20260925 14:57:37.953581 7575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 14:57:37 dist-test-slave-v5vl krb5kdc[7495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:37.966259 7575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3: Bootstrap starting.
I20260925 14:57:37.970487 7575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:37.971757 7575 log.cc:824] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:37.974895 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:37.952430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53663 (local address 127.17.133.254:39909)
0925 14:57:37.953275 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:37.953290 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:37.953340 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:57:37.955644 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:37.955671 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:37.956169 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:37.956640 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:37.957949 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:37.960019 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:37.961311 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:37.962321 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:37.962501 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:37.966395 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:37.966406 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:37.966425 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:37.966472 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:37.970233 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:37.970956 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:37.970964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:37.970972 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:37.971160 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:37.971552 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:37.971558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:37.971561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:37.972417 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:37.972758 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:37.973193 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 14:57:37.973611 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":395,"threads_started":1}
I20260925 14:57:37.975102 7575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3: No bootstrap required, opened a new log
I20260925 14:57:37.978663 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:37.983959 7575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd009d274cb42c29470f2db78702db3" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39909 } }
I20260925 14:57:37.984324 7575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:37.984441 7575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd009d274cb42c29470f2db78702db3, State: Initialized, Role: FOLLOWER
I20260925 14:57:37.985013 7575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd009d274cb42c29470f2db78702db3" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39909 } }
I20260925 14:57:37.985252 7575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:37.985409 7575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:37.985610 7575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:37.987906 7575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd009d274cb42c29470f2db78702db3" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39909 } }
I20260925 14:57:37.988396 7575 leader_election.cc:302] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fd009d274cb42c29470f2db78702db3; no voters:
I20260925 14:57:37.989018 7575 leader_election.cc:288] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:37.989084 7585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:37.991123 7585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [term 1 LEADER]: Becoming Leader. State: Replica: 5fd009d274cb42c29470f2db78702db3, State: Running, Role: LEADER
I20260925 14:57:37.991312 7575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:37.991797 7585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd009d274cb42c29470f2db78702db3" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39909 } }
I20260925 14:57:37.994679 7587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd009d274cb42c29470f2db78702db3. Latest consensus state: current_term: 1 leader_uuid: "5fd009d274cb42c29470f2db78702db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd009d274cb42c29470f2db78702db3" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39909 } } }
I20260925 14:57:37.995071 7587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:38.000159 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:37.982744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45725 (local address 127.17.133.254:39909)
0925 14:57:37.982893 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:37.982897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:37.983481 (+ 584us) server_negotiation.cc:407] Connection header received
0925 14:57:37.983609 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:37.983614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:37.983699 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:37.983797 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:37.985427 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:37.986305 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:37.987531 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:37.987948 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:37.988079 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:37.990995 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:37.991004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:37.991012 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:37.991057 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:37.995192 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:37.996204 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:37.996212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:37.996219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:37.996419 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:37.996992 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:37.997000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:37.997006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:37.997522 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:37.999331 (+ 1809us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:37.999544 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 14:57:37.999828 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:57:38.001622 7586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fd009d274cb42c29470f2db78702db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd009d274cb42c29470f2db78702db3" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 39909 } } }
I20260925 14:57:38.001931 7586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:38.004993 7591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:38.010257 7591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:38.018883 7591 catalog_manager.cc:1347] Generated new cluster ID: 09c52f52206e41f99206efd33225835c
I20260925 14:57:38.019007 7591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:38.040495 7591 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:38.041877 7591 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:38.051383 7591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fd009d274cb42c29470f2db78702db3: Generated new TSK 0
I20260925 14:57:38.052430 7591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:38.063356 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7508
I20260925 14:57:38.064662 7570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:38.156903 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7508
2026-09-25T14:57:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:38 dist-test-slave-v5vl krb5kdc[7608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:38 dist-test-slave-v5vl krb5kdc[7608](info): set up 2 sockets
Sep 25 14:57:38 dist-test-slave-v5vl krb5kdc[7608](info): commencing operation
krb5kdc: starting...
W20260925 14:57:39.261262 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:39 dist-test-slave-v5vl krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:57:39.511435 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:44359
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:44359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:39.842837 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:39.843137 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:39.843238 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:39.850322 7621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:39.850438 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:39.850483 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:39.850544 7621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:39.850582 7621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:39.860240 7621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:44359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:44359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:39.861581 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:39.862834 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:39.874559 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:39.874576 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:39.874529 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:39.875232 7621 server_base.cc:1034] running on GCE node
I20260925 14:57:39.876082 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:39.877573 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:39.878815 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790348259878763 us; error 50 us; skew 500 ppm
I20260925 14:57:39.879180 7621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:39 dist-test-slave-v5vl krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348259, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:39.882337 7621 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:57:39.883741 7621 webserver.cc:460] Webserver started at http://127.17.133.254:38839/ using document root <none> and password file <none>
I20260925 14:57:39.884438 7621 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:39.884631 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:39.884956 7621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:39.887918 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "c0eea606f0754ae89bf750b1149bce7c"
format_stamp: "Formatted at 2026-09-25 14:57:39 on dist-test-slave-v5vl"
server_key: "119ffc80e0592e6a1392b34243fe1b9f"
server_key_iv: "e68ab9c0a65c792d7bef3b5b47cd5120"
server_key_version: "encryptionkey@0"
I20260925 14:57:39.888760 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "c0eea606f0754ae89bf750b1149bce7c"
format_stamp: "Formatted at 2026-09-25 14:57:39 on dist-test-slave-v5vl"
server_key: "119ffc80e0592e6a1392b34243fe1b9f"
server_key_iv: "e68ab9c0a65c792d7bef3b5b47cd5120"
server_key_version: "encryptionkey@0"
I20260925 14:57:39.893708 7621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 14:57:39.897011 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:39.898270 7621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 14:57:39.898451 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "c0eea606f0754ae89bf750b1149bce7c"
format_stamp: "Formatted at 2026-09-25 14:57:39 on dist-test-slave-v5vl"
server_key: "119ffc80e0592e6a1392b34243fe1b9f"
server_key_iv: "e68ab9c0a65c792d7bef3b5b47cd5120"
server_key_version: "encryptionkey@0"
I20260925 14:57:39.898643 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:39.911612 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:39.915784 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:39.916070 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:39.934878 7621 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:44359
I20260925 14:57:39.934901 7687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:44359 every 8 connection(s)
I20260925 14:57:39.936372 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:39.940194 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:39.946075 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7621
I20260925 14:57:39.946250 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:39.946684 17943 external_mini_cluster.cc:1428] Setting key 3bb5d6aaca73044039b8996869d431b5
I20260925 14:57:39.951859 7688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c: Bootstrap starting.
I20260925 14:57:39.955850 7688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:39.956948 7688 log.cc:824] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:39.960028 7688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c: No bootstrap required, opened a new log
I20260925 14:57:39.961685 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:39.948641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60645 (local address 127.17.133.254:44359)
0925 14:57:39.949500 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:39.949516 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:39.949567 (+ 51us) server_negotiation.cc:407] Connection header received
0925 14:57:39.951942 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:39.951971 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:39.952484 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:39.952959 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:39.953486 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:39.955524 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:39.956999 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:39.957962 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:39.958207 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:39.959395 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:39.959402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:39.959417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:39.959948 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:39.960025 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:39.960049 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:39.960179 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:39.960399 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 14:57:39.960718 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":355,"threads_started":1}
I20260925 14:57:39.965512 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:39.968354 7688 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eea606f0754ae89bf750b1149bce7c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44359 } }
I20260925 14:57:39.968796 7688 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:39.968904 7688 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0eea606f0754ae89bf750b1149bce7c, State: Initialized, Role: FOLLOWER
I20260925 14:57:39.969543 7688 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eea606f0754ae89bf750b1149bce7c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44359 } }
I20260925 14:57:39.969836 7688 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:39.969973 7688 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:39.970149 7688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:39.973201 7688 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eea606f0754ae89bf750b1149bce7c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44359 } }
I20260925 14:57:39.973793 7688 leader_election.cc:302] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0eea606f0754ae89bf750b1149bce7c; no voters:
I20260925 14:57:39.974501 7688 leader_election.cc:288] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:39.974645 7698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:39.975503 7698 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [term 1 LEADER]: Becoming Leader. State: Replica: c0eea606f0754ae89bf750b1149bce7c, State: Running, Role: LEADER
I20260925 14:57:39.976241 7698 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eea606f0754ae89bf750b1149bce7c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44359 } }
I20260925 14:57:39.976702 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:39.969638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35023 (local address 127.17.133.254:44359)
0925 14:57:39.969821 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:39.969832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:39.970398 (+ 566us) server_negotiation.cc:407] Connection header received
0925 14:57:39.970536 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:39.970544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:39.970632 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:39.970751 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:39.971168 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:39.972265 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:39.973667 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:39.974420 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:39.974577 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:39.975783 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:39.975788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:39.975793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:39.975814 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:39.975860 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:39.975886 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:39.975975 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:39.976224 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 14:57:39.976324 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:57:39.977056 7688 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:39.980837 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0eea606f0754ae89bf750b1149bce7c. Latest consensus state: current_term: 1 leader_uuid: "c0eea606f0754ae89bf750b1149bce7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eea606f0754ae89bf750b1149bce7c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44359 } } }
I20260925 14:57:39.981222 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:39.981769 7699 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0eea606f0754ae89bf750b1149bce7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eea606f0754ae89bf750b1149bce7c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 44359 } } }
I20260925 14:57:39.982088 7699 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:39.987811 7708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:39.993243 7708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:40.002614 7708 catalog_manager.cc:1347] Generated new cluster ID: 1a95134dea95423a9fdb0b8eed1bd3c1
I20260925 14:57:40.002736 7708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:40.016989 7708 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:40.018326 7708 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:40.025331 7708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0eea606f0754ae89bf750b1149bce7c: Generated new TSK 0
I20260925 14:57:40.026192 7708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:40.031991 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7621
W20260925 14:57:40.033351 7683 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60645
W20260925 14:57:40.034096 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7621: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:40.034248 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7621
2026-09-25T14:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:40 dist-test-slave-v5vl krb5kdc[7719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:40 dist-test-slave-v5vl krb5kdc[7719](info): set up 2 sockets
Sep 25 14:57:40 dist-test-slave-v5vl krb5kdc[7719](info): commencing operation
krb5kdc: starting...
W20260925 14:57:41.129215 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:41 dist-test-slave-v5vl krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:57:41.378217 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46449
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:46449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:41.703017 7732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:41.703315 7732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:41.703513 7732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:41.710639 7732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:41.710739 7732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:41.710834 7732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:41.710938 7732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:41.711023 7732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:41.720487 7732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:46449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:41.721784 7732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:41.723124 7732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:41.735163 7737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:41.735124 7740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:41.735076 7738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:41.736003 7732 server_base.cc:1034] running on GCE node
I20260925 14:57:41.736934 7732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:41.738372 7732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:41.739614 7732 hybrid_clock.cc:648] HybridClock initialized: now 1790348261739577 us; error 31 us; skew 500 ppm
I20260925 14:57:41.740069 7732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:41 dist-test-slave-v5vl krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348261, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:41.743271 7732 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:57:41.744836 7732 webserver.cc:460] Webserver started at http://127.17.133.254:43571/ using document root <none> and password file <none>
I20260925 14:57:41.745509 7732 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:41.745651 7732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:41.745956 7732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:41.749050 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "9324e0beb845423b99c9ac37d670847f"
format_stamp: "Formatted at 2026-09-25 14:57:41 on dist-test-slave-v5vl"
server_key: "3a26af095039bb0d6eb0684054940f6a"
server_key_iv: "239b9321c826bcd74540459462a0aee3"
server_key_version: "encryptionkey@0"
I20260925 14:57:41.749871 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "9324e0beb845423b99c9ac37d670847f"
format_stamp: "Formatted at 2026-09-25 14:57:41 on dist-test-slave-v5vl"
server_key: "3a26af095039bb0d6eb0684054940f6a"
server_key_iv: "239b9321c826bcd74540459462a0aee3"
server_key_version: "encryptionkey@0"
I20260925 14:57:41.755440 7732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 14:57:41.759233 7747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:41.760680 7732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 14:57:41.760871 7732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "9324e0beb845423b99c9ac37d670847f"
format_stamp: "Formatted at 2026-09-25 14:57:41 on dist-test-slave-v5vl"
server_key: "3a26af095039bb0d6eb0684054940f6a"
server_key_iv: "239b9321c826bcd74540459462a0aee3"
server_key_version: "encryptionkey@0"
I20260925 14:57:41.761080 7732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:41.766995 7732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:41.771121 7732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:41.771420 7732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:41.790050 7732 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:46449
I20260925 14:57:41.790082 7798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:46449 every 8 connection(s)
I20260925 14:57:41.791530 7732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:41.795270 7799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:41.801587 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7732
I20260925 14:57:41.801759 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:41.802647 17943 external_mini_cluster.cc:1428] Setting key 100c85237a139127449a426a7ebe2540
I20260925 14:57:41.807237 7799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f: Bootstrap starting.
I20260925 14:57:41.811475 7799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:41.812906 7799 log.cc:824] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:41.815711 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:41.804969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35631 (local address 127.17.133.254:46449)
0925 14:57:41.807814 (+ 2845us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:41.807831 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:41.807886 (+ 55us) server_negotiation.cc:407] Connection header received
0925 14:57:41.810404 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:41.810435 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:41.810885 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:41.811331 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:41.813420 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:41.813437 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:41.813460 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:41.813488 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:41.813581 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:41.813614 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:41.813860 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:41.814136 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 14:57:41.814507 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2579,"thread_start_us":2314,"threads_started":1}
I20260925 14:57:41.816673 7799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f: No bootstrap required, opened a new log
I20260925 14:57:41.820621 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:41.827504 7799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9324e0beb845423b99c9ac37d670847f" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46449 } }
I20260925 14:57:41.828063 7799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:41.828222 7799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9324e0beb845423b99c9ac37d670847f, State: Initialized, Role: FOLLOWER
I20260925 14:57:41.828768 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:41.825587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44285 (local address 127.17.133.254:46449)
0925 14:57:41.825784 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:41.825791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:41.826416 (+ 625us) server_negotiation.cc:407] Connection header received
0925 14:57:41.826487 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:41.826496 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:41.826570 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:41.826687 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:41.827977 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:41.827981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:41.827985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:41.828004 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:41.828053 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:41.828078 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:41.828229 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:41.828355 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 14:57:41.828419 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:57:41.829011 7799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9324e0beb845423b99c9ac37d670847f" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46449 } }
I20260925 14:57:41.829342 7799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:41.829499 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:41.829665 7799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:41.832472 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9324e0beb845423b99c9ac37d670847f" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46449 } }
I20260925 14:57:41.832995 7799 leader_election.cc:302] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9324e0beb845423b99c9ac37d670847f; no voters:
I20260925 14:57:41.833624 7799 leader_election.cc:288] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:41.833734 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:41.834543 7809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [term 1 LEADER]: Becoming Leader. State: Replica: 9324e0beb845423b99c9ac37d670847f, State: Running, Role: LEADER
I20260925 14:57:41.835186 7809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9324e0beb845423b99c9ac37d670847f" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46449 } }
I20260925 14:57:41.835722 7799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:41.838564 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9324e0beb845423b99c9ac37d670847f. Latest consensus state: current_term: 1 leader_uuid: "9324e0beb845423b99c9ac37d670847f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9324e0beb845423b99c9ac37d670847f" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46449 } } }
I20260925 14:57:41.839090 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:41.839560 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9324e0beb845423b99c9ac37d670847f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9324e0beb845423b99c9ac37d670847f" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 46449 } } }
I20260925 14:57:41.839843 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f [sys.catalog]: This master's current role is: LEADER
W20260925 14:57:41.851064 7825 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:41.851230 7825 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:41.851369 7818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:41.853499 7818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:41.863734 7818 catalog_manager.cc:1347] Generated new cluster ID: 2d64c02b645e4cdbb729c5b3c679b8ad
I20260925 14:57:41.863857 7818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:41.888150 7818 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:41.888641 7818 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:41.895748 7818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9324e0beb845423b99c9ac37d670847f: Generated new TSK 0
I20260925 14:57:41.896768 7818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:41.911845 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7732
W20260925 14:57:41.913666 7794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35631
W20260925 14:57:41.914371 17943 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7732: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:57:41.914529 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7732
2026-09-25T14:57:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1880 ms)
[----------] 18 tests from AuthTokenIssuingTest (34294 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:42 dist-test-slave-v5vl krb5kdc[7830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:42 dist-test-slave-v5vl krb5kdc[7830](info): set up 2 sockets
Sep 25 14:57:42 dist-test-slave-v5vl krb5kdc[7830](info): commencing operation
krb5kdc: starting...
W20260925 14:57:43.009327 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:43 dist-test-slave-v5vl krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:57:43.267877 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:40045
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:40045,127.17.133.253:37257,127.17.133.252:35001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:43.597752 7843 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:43.598170 7843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:43.598296 7843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:43.598445 7843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:43.605949 7843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:43.606050 7843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:43.606148 7843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:43.606237 7843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:43.606350 7843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:43.616928 7843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:40045,127.17.133.253:37257,127.17.133.252:35001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:40045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:43.618363 7843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:43.619725 7843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:43.631723 7848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.631753 7849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.631685 7851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:43.632768 7843 server_base.cc:1034] running on GCE node
I20260925 14:57:43.633460 7843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:43.634761 7843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:43.635996 7843 hybrid_clock.cc:648] HybridClock initialized: now 1790348263635926 us; error 50 us; skew 500 ppm
I20260925 14:57:43.636309 7843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:43 dist-test-slave-v5vl krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:43.642421 7843 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:57:43.643853 7843 webserver.cc:460] Webserver started at http://127.17.133.254:40243/ using document root <none> and password file <none>
I20260925 14:57:43.644543 7843 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:43.644711 7843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:43.644990 7843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:43.647889 7843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "1cff622f4d174add86a41f3f68d93654"
format_stamp: "Formatted at 2026-09-25 14:57:43 on dist-test-slave-v5vl"
server_key: "15ba1dab8b6af1450161eac2551194aa"
server_key_iv: "7e4ae8eb1f1a443003b7b3837dd31bca"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.648674 7843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "1cff622f4d174add86a41f3f68d93654"
format_stamp: "Formatted at 2026-09-25 14:57:43 on dist-test-slave-v5vl"
server_key: "15ba1dab8b6af1450161eac2551194aa"
server_key_iv: "7e4ae8eb1f1a443003b7b3837dd31bca"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.653594 7843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 14:57:43.656749 7858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.657989 7843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:57:43.658174 7843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "1cff622f4d174add86a41f3f68d93654"
format_stamp: "Formatted at 2026-09-25 14:57:43 on dist-test-slave-v5vl"
server_key: "15ba1dab8b6af1450161eac2551194aa"
server_key_iv: "7e4ae8eb1f1a443003b7b3837dd31bca"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.658404 7843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:43.678969 7843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:43.682920 7843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:43.683194 7843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:43.701532 7843 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:40045
I20260925 14:57:43.701545 7909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:40045 every 8 connection(s)
I20260925 14:57:43.702915 7843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:43.703488 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7843
I20260925 14:57:43.703720 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:43.704594 17943 external_mini_cluster.cc:1428] Setting key 3f903781a140db6f2b4bc0e87f3bbe80
I20260925 14:57:43.706873 7910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:43.713430 7910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.133.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.253@KRBTEST.COM" created.
Sep 25 14:57:43 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:43.740263 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.716559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:55916)
0925 14:57:43.717529 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.717581 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.720506 (+ 2925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:43.721470 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:43.721480 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:43.722342 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:43.722869 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.722898 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.724608 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.724615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.725681 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.725692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.726612 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.726619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.726792 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.727508 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:43.727530 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:43.731792 (+ 4262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:43.736204 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:43.736232 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:43.736242 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:43.736690 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:43.737145 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:43.737148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:43.737167 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:43.737315 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:43.738338 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:43.738344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:43.738549 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 14:57:43.738785 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 14:57:43.739192 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":382,"threads_started":1}
I20260925 14:57:43.740311 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.716559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.17.133.254:40045)
0925 14:57:43.718487 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.718493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.718535 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:57:43.720956 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.720963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.721214 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.721392 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:43.723212 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.724429 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.725820 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.726381 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.726583 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.732068 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:43.732082 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:43.732104 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:43.732155 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:43.736043 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.736826 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.736837 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.736840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.737032 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.737430 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.737437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.737443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.738196 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:43.738410 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.738987 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.739401 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":1703,"spinlock_wait_cycles":65792,"thread_start_us":1477,"threads_started":1}
I20260925 14:57:43.743362 7910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:43.744410 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.744038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48502)
0925 14:57:43.744197 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.744300 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 14:57:43.744769 7859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
I20260925 14:57:43.746255 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.745926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48504)
0925 14:57:43.746063 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.746140 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 14:57:43.746541 7910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:37257: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab.
Entry for principal kudu/127.17.133.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab.
I20260925 14:57:43.763929 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.253:37257
--webserver_interface=127.17.133.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab
--principal=kudu/127.17.133.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:40045,127.17.133.253:37257,127.17.133.252:35001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:43.795971 7910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } attempt: 1
I20260925 14:57:43.797369 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.796919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48518)
0925 14:57:43.797127 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.797231 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 14:57:43.798561 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.798246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48524)
0925 14:57:43.798381 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.798444 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 14:57:43.798806 7910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:37257: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
I20260925 14:57:43.867067 7910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } attempt: 2
I20260925 14:57:43.868283 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.867822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48528)
0925 14:57:43.868058 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.868149 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 14:57:43.869370 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.869045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48530)
0925 14:57:43.869174 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.869252 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 14:57:43.869606 7910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:37257: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
I20260925 14:57:43.960929 7910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } attempt: 3
I20260925 14:57:43.962195 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.961721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48532)
0925 14:57:43.961921 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.962047 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 14:57:43.963330 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.962962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48534)
0925 14:57:43.963125 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.963199 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 14:57:43.963591 7910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:37257: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
I20260925 14:57:44.106884 7910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } attempt: 4
I20260925 14:57:44.108168 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.107641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48536)
0925 14:57:44.107948 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.108034 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260925 14:57:44.109302 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.108958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48544)
0925 14:57:44.109094 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.109180 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 14:57:44.109535 7910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:37257: Network error: Client connection negotiation failed: client connection to 127.17.133.253:37257: connect: Connection refused (error 111)
W20260925 14:57:44.125670 7917 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:44.126044 7917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:44.126145 7917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:44.126243 7917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:44.133589 7917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:44.133682 7917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:44.133792 7917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:44.133869 7917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:44.133961 7917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:44.144016 7917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:40045,127.17.133.253:37257,127.17.133.252:35001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab
--principal=kudu/127.17.133.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.253:37257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.133.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:44.145397 7917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:44.146719 7917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:44.158792 7923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.158722 7925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.158829 7922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:44.159729 7917 server_base.cc:1034] running on GCE node
I20260925 14:57:44.160523 7917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:44.161944 7917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:44.163173 7917 hybrid_clock.cc:648] HybridClock initialized: now 1790348264163099 us; error 52 us; skew 500 ppm
I20260925 14:57:44.163496 7917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:44.166653 7917 init.cc:377] Logged in from keytab as kudu/127.17.133.253@KRBTEST.COM (short username kudu)
I20260925 14:57:44.168102 7917 webserver.cc:460] Webserver started at http://127.17.133.253:33569/ using document root <none> and password file <none>
I20260925 14:57:44.168860 7917 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:44.169013 7917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:44.169425 7917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:44.172324 7917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/data/instance:
uuid: "037c948e1b6e4878aeb3e3da7f906c2e"
format_stamp: "Formatted at 2026-09-25 14:57:44 on dist-test-slave-v5vl"
server_key: "5c9b2bbb4510bc1570f1e1e471c2c946"
server_key_iv: "331b2dc7fc8199f3e37d2f0e36f68d0f"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.173125 7917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/wal/instance:
uuid: "037c948e1b6e4878aeb3e3da7f906c2e"
format_stamp: "Formatted at 2026-09-25 14:57:44 on dist-test-slave-v5vl"
server_key: "5c9b2bbb4510bc1570f1e1e471c2c946"
server_key_iv: "331b2dc7fc8199f3e37d2f0e36f68d0f"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.179212 7917 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 14:57:44.182786 7932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.184124 7917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:57:44.184300 7917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/wal
uuid: "037c948e1b6e4878aeb3e3da7f906c2e"
format_stamp: "Formatted at 2026-09-25 14:57:44 on dist-test-slave-v5vl"
server_key: "5c9b2bbb4510bc1570f1e1e471c2c946"
server_key_iv: "331b2dc7fc8199f3e37d2f0e36f68d0f"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.184518 7917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:44.203913 7917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:44.208400 7917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:44.208701 7917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:44.227108 7917 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.253:37257
I20260925 14:57:44.227121 7983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.253:37257 every 8 connection(s)
I20260925 14:57:44.228504 7917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
I20260925 14:57:44.229764 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7917
I20260925 14:57:44.229959 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-1/wal/instance
I20260925 14:57:44.230542 17943 external_mini_cluster.cc:1428] Setting key 76b101916f3a963f5adbcbce5be8e36c
I20260925 14:57:44.233208 7984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:44.242580 7984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.133.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.252@KRBTEST.COM" created.
I20260925 14:57:44.263528 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.245192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.17.133.254:40045)
0925 14:57:44.245643 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.245647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.246640 (+ 993us) server_negotiation.cc:407] Connection header received
0925 14:57:44.248446 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.248453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.248549 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.248713 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.250253 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.251118 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.252651 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.253055 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.253203 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.256810 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.256814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.256817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.256868 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.260095 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.260899 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.260907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.260913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.261064 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.261466 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.261473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.261479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.261939 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.262089 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.262808 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.263055 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
I20260925 14:57:44.264309 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.245785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:55926)
0925 14:57:44.246476 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.246515 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.248157 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.248939 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.248947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.249455 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:44.250048 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.250065 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.251275 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.251279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.252482 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.252494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.253276 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.253283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.253522 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.254223 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:44.254244 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:44.256542 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:44.260258 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.260266 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.260280 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.260784 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.261172 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.261175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.261177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.261348 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.262085 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:44.262091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:44.262426 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.262653 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.263187 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":293,"threads_started":1}
I20260925 14:57:44.266170 7984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
I20260925 14:57:44.283510 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.267010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48556)
0925 14:57:44.267243 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.267264 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.267429 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.268722 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.268725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.269094 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:44.269230 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.269237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.271532 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.271536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.272444 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.272456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.273671 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.273675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.273853 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.274498 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:44.274516 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:44.276535 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:44.281054 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.281058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.281061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.281449 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.281882 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.281886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.281888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.282026 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.282803 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:44.282806 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:44.282917 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.283022 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.283237 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":22}
I20260925 14:57:44.284039 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.267482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.17.133.253:37257)
0925 14:57:44.268197 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.268208 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.268243 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:57:44.268301 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.268310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.268554 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.268732 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.269405 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.271207 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.272612 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.273535 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.273672 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.276832 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.276856 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.276876 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.276924 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.280909 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.281589 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.281601 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.281606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.281758 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.282142 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.282148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.282153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.282669 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.282915 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.283228 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.283538 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":529,"spinlock_wait_cycles":24960,"thread_start_us":389,"threads_started":1}
I20260925 14:57:44.286386 7984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:44.287472 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.287055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45540)
0925 14:57:44.287229 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.287328 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 14:57:44.287738 7934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
I20260925 14:57:44.289381 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.289028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45554)
0925 14:57:44.289210 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.289283 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 14:57:44.289578 7984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:35001: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab.
Entry for principal kudu/127.17.133.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab.
I20260925 14:57:44.293633 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.252:35001
--webserver_interface=127.17.133.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab
--principal=kudu/127.17.133.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:40045,127.17.133.253:37257,127.17.133.252:35001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:44.338887 7984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } attempt: 1
I20260925 14:57:44.340099 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.339639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45564)
0925 14:57:44.339848 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.339953 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 14:57:44.341116 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.340781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45580)
0925 14:57:44.340942 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.341017 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 14:57:44.341393 7984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:35001: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
I20260925 14:57:44.408895 7910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } attempt: 5
I20260925 14:57:44.409631 7984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } attempt: 2
I20260925 14:57:44.410640 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.410196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45592)
0925 14:57:44.410346 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.410437 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 14:57:44.411803 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.411361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45594)
0925 14:57:44.411521 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.411612 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 14:57:44.412120 7984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:35001: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
I20260925 14:57:44.423132 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.409816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.17.133.253:37257)
0925 14:57:44.410055 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.410062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.410135 (+ 73us) server_negotiation.cc:407] Connection header received
0925 14:57:44.410368 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.410374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.410453 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.410549 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.411535 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.412519 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.413729 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.414197 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.414328 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.417948 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.417953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.417956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.417989 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.420648 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.421412 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.421416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.421418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.421527 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.421916 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.421919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.421921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.422175 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.422279 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.422762 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.422930 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 14:57:44.423257 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.409739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48564)
0925 14:57:44.410029 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.410049 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.410196 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.410624 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.410630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.411098 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:44.411384 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.411400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.412677 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.412683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.413577 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.413588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.414362 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.414369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.414575 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.415199 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:44.415217 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:44.417759 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:44.420806 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.420811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.420815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.421305 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.421628 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.421633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.421637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.421817 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.422283 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:44.422287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:44.422461 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.422617 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.422930 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260925 14:57:44.424242 7910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:44.425258 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.424922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45608)
0925 14:57:44.425074 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.425144 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 14:57:44.426139 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.425824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45624)
0925 14:57:44.425960 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.426044 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 14:57:44.426358 7910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:35001: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
I20260925 14:57:44.477667 7910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } attempt: 1
I20260925 14:57:44.478816 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.478356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45626)
0925 14:57:44.478555 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.478656 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 14:57:44.479758 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.479459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45642)
0925 14:57:44.479594 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.479663 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 14:57:44.479959 7910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:35001: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
I20260925 14:57:44.503372 7984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } attempt: 3
I20260925 14:57:44.504751 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.504158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45658)
0925 14:57:44.504375 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.504500 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 14:57:44.505833 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.505381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45668)
0925 14:57:44.505596 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.505693 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260925 14:57:44.506047 7984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:35001: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
I20260925 14:57:44.548184 7910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } attempt: 2
I20260925 14:57:44.549387 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.548892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45672)
0925 14:57:44.549128 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.549223 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 14:57:44.550576 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.550115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45686)
0925 14:57:44.550274 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.550394 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 14:57:44.550814 7910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:35001: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
W20260925 14:57:44.624364 7992 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:44.624688 7992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:44.624754 7992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:44.624835 7992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:44.631902 7992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:44.631990 7992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:44.632047 7992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:44.632098 7992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:44.632153 7992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:44.641798 7992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:40045,127.17.133.253:37257,127.17.133.252:35001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab
--principal=kudu/127.17.133.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.252:35001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.133.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:44.643030 7992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:44.644223 7992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:44.649284 7984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } attempt: 4
I20260925 14:57:44.650588 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.650056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45702)
0925 14:57:44.650328 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.650445 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 14:57:44.651458 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.651184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45714)
0925 14:57:44.651304 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.651367 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 14:57:44.651741 7984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:35001: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
W20260925 14:57:44.656262 7997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.656296 7998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.656664 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:44.657006 7910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } attempt: 3
I20260925 14:57:44.657300 7992 server_base.cc:1034] running on GCE node
I20260925 14:57:44.658088 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.657630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45724)
0925 14:57:44.657827 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.657904 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 14:57:44.658217 7992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:44.659102 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.658726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45730)
0925 14:57:44.658922 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.659005 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260925 14:57:44.659330 7910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:35001: Network error: Client connection negotiation failed: client connection to 127.17.133.252:35001: connect: Connection refused (error 111)
I20260925 14:57:44.659680 7992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:44.660909 7992 hybrid_clock.cc:648] HybridClock initialized: now 1790348264660849 us; error 30 us; skew 500 ppm
I20260925 14:57:44.661239 7992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:44.664301 7992 init.cc:377] Logged in from keytab as kudu/127.17.133.252@KRBTEST.COM (short username kudu)
I20260925 14:57:44.665771 7992 webserver.cc:460] Webserver started at http://127.17.133.252:45399/ using document root <none> and password file <none>
I20260925 14:57:44.666455 7992 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:44.666582 7992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:44.666888 7992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:44.669796 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/data/instance:
uuid: "f8bdd18f33264c138aced249f833e65e"
format_stamp: "Formatted at 2026-09-25 14:57:44 on dist-test-slave-v5vl"
server_key: "abd07077bdef07fe38a2f556e6190b16"
server_key_iv: "3a6335eb6b65c7ab70e0fb286fb3e3df"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.670563 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/wal/instance:
uuid: "f8bdd18f33264c138aced249f833e65e"
format_stamp: "Formatted at 2026-09-25 14:57:44 on dist-test-slave-v5vl"
server_key: "abd07077bdef07fe38a2f556e6190b16"
server_key_iv: "3a6335eb6b65c7ab70e0fb286fb3e3df"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.675458 7992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 14:57:44.678714 8007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.679955 7992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:57:44.680124 7992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/wal
uuid: "f8bdd18f33264c138aced249f833e65e"
format_stamp: "Formatted at 2026-09-25 14:57:44 on dist-test-slave-v5vl"
server_key: "abd07077bdef07fe38a2f556e6190b16"
server_key_iv: "3a6335eb6b65c7ab70e0fb286fb3e3df"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.680325 7992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:44.694957 7992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:44.698678 7992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:44.698941 7992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:44.717258 7992 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.252:35001
I20260925 14:57:44.717278 8058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.252:35001 every 8 connection(s)
I20260925 14:57:44.718541 7992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
I20260925 14:57:44.722002 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:44.727919 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 7992
I20260925 14:57:44.728096 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347918228911-17943-0/minicluster-data/master-2/wal/instance
I20260925 14:57:44.728521 17943 external_mini_cluster.cc:1428] Setting key 81fa5a5d97c52dd41288df7ccc33213c
I20260925 14:57:44.727968 8059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
I20260925 14:57:44.750686 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.730681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.17.133.254:40045)
0925 14:57:44.730923 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.730928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.731744 (+ 816us) server_negotiation.cc:407] Connection header received
0925 14:57:44.733038 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.733046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.733134 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.733237 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.734893 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.735978 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.738120 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.738895 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.739038 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.743834 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.743839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.743842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.743894 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.747215 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.748149 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.748152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.748154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.748259 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.748723 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.748729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.748732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.749123 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.749260 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.750063 (+ 803us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.750310 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260925 14:57:44.752014 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.730286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.17.133.252:35001)
0925 14:57:44.731104 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.731113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.731149 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:57:44.732804 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.732823 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.733083 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.733269 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.734957 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.736631 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.738395 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.738891 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.739073 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.744758 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.744775 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.744794 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.744845 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.748330 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.749149 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.749156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.749159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.749303 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.749652 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.749656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.749659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.750032 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.750283 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.750652 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.750896 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":433,"threads_started":1}
I20260925 14:57:44.752108 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.730803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:40045 (local address 127.0.0.1:55932)
0925 14:57:44.731581 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.731619 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.732813 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.733305 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.733312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.734146 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:44.734680 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.734708 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.736159 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.736164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.737957 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.737972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.739356 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.739366 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.739565 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.740181 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:44.740232 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:44.743656 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:44.747336 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.747351 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.747361 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.748037 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.748350 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.748354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.748357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.748577 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.749252 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:44.749259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:44.749680 (+ 421us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.749967 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.750513 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":402,"threads_started":1}
I20260925 14:57:44.753633 8059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:44.767185 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.754292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:37257 (local address 127.0.0.1:48576)
0925 14:57:44.754456 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.754477 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.754574 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.754870 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.754874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.755305 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:44.755512 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.755527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.756645 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.756653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.757740 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.757751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.758474 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.758479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.758662 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.759223 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:44.759240 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:44.761266 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:44.764866 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.764875 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.764882 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.765425 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.765829 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.765833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.765836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.765969 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.766545 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:44.766549 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:44.766668 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.766774 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.766960 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 14:57:44.767692 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.754404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.17.133.253:37257)
0925 14:57:44.754585 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.754590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.754610 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:57:44.754690 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.754694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.754751 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.754859 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.755663 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.756496 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.757876 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.758334 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.758466 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.761470 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.761477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.761482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.761523 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.764746 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.765553 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.765557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.765562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.765719 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.766069 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.766074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.766077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.766415 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.766574 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.766892 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.767113 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:57:44.768646 8059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:44.771698 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.755738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.17.133.254:40045)
0925 14:57:44.755912 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.755917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.756838 (+ 921us) server_negotiation.cc:407] Connection header received
0925 14:57:44.756929 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.756936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.757018 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.757115 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.758175 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.758870 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.760077 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.760486 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.760654 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.765647 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.765656 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.765660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.765696 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.768985 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.769755 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.769767 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.769772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.769926 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.770323 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.770326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.770328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.770603 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.770954 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.771162 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.771369 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
I20260925 14:57:44.783282 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.769379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45744)
0925 14:57:44.769587 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.769606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.769699 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.770056 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.770061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.770527 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:44.770689 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.770701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.772040 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.772043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.773329 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.773344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.774316 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.774319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.774458 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.775010 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:44.775033 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:44.777318 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:44.780843 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.780847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.780849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.781171 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.781538 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.781542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.781546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.781721 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.782224 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:44.782228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:44.782364 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.782540 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.782833 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 14:57:44.783367 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.769584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.17.133.252:35001)
0925 14:57:44.769739 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.769744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.769761 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:57:44.769805 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.769810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.769878 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.769980 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.770873 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.771905 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.773472 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.774083 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.774173 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.777482 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.777486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.777489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.777515 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.780706 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.781288 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.781295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.781300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.781427 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.781837 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.781841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.781845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.782109 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.782259 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.782635 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.782962 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":53}
I20260925 14:57:44.787774 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.773512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.17.133.253:37257)
0925 14:57:44.773686 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.773692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.773708 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:57:44.773753 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.773757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.773840 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.773932 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.774981 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.776054 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.777300 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.777759 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.777844 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.782478 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.782486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.782494 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.782533 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.785365 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.785937 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.785944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.785947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.786072 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.786424 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.786429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.786433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.786669 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.786922 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.787185 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.787428 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:57:44.788318 8059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e: Bootstrap starting.
I20260925 14:57:44.792459 8059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:44.792677 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:44.793716 8059 log.cc:824] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:44.796406 8059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e: No bootstrap required, opened a new log
I20260925 14:57:44.795746 7885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1cff622f4d174add86a41f3f68d93654"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55944
I20260925 14:57:44.803436 8059 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } }
I20260925 14:57:44.803769 8059 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8bdd18f33264c138aced249f833e65e, State: Initialized, Role: FOLLOWER
I20260925 14:57:44.804217 8059 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } }
I20260925 14:57:44.805176 8065 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:44.805537 8065 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:44.806133 8059 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [sys.catalog]: configured and running, proceeding with master startup.
W20260925 14:57:44.815578 8076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:44.815706 8076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:44.813411 7992 master_runner.cc:186] Error getting master registration for 127.17.133.254:40045: OK, instance_id { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" instance_seqno: 1790348263696223 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:44.824470 7992 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:57:44.836549 7910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } attempt: 4
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
I20260925 14:57:44.851649 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.837287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45758)
0925 14:57:44.837561 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.837581 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.837705 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.838070 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.838076 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.838523 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:44.838688 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.838697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.839752 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.839758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.840829 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.840842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.841602 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.841607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.841817 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.842481 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:44.842499 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:44.844994 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:44.848853 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.848859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.848863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.849374 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.849786 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.849793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.849799 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.849966 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.850627 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:44.850632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:44.850812 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.850965 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.851258 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 14:57:44.851658 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.837490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.17.133.252:35001)
0925 14:57:44.837696 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.837702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.837724 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:57:44.837860 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.837865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.837937 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.838026 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.838854 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.839599 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.840971 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.841451 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.841589 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.845181 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.845188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.845193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.845234 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.848715 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.849508 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.849514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.849518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.849669 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.850072 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.850078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.850081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.850450 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.850648 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.851077 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.851289 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:57:44.858038 7910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654: Bootstrap starting.
I20260925 14:57:44.862246 7910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:44.863574 7910 log.cc:824] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:44.867146 7910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654: No bootstrap required, opened a new log
I20260925 14:57:44.875027 7910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } }
I20260925 14:57:44.875478 7910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cff622f4d174add86a41f3f68d93654, State: Initialized, Role: FOLLOWER
I20260925 14:57:44.876011 7910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } }
I20260925 14:57:44.877149 8079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:44.877533 8079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:44.878479 7910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:44.878546 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:57:44.878770 7885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:44.881620 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } }
I20260925 14:57:44.883126 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [CANDIDATE]: Term 1 election: Requested vote from peers 037c948e1b6e4878aeb3e3da7f906c2e (127.17.133.253:37257), f8bdd18f33264c138aced249f833e65e (127.17.133.252:35001)
I20260925 14:57:44.884137 8034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cff622f4d174add86a41f3f68d93654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f8bdd18f33264c138aced249f833e65e"
I20260925 14:57:44.884704 8034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:44.891112 8034 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cff622f4d174add86a41f3f68d93654 in term 1.
I20260925 14:57:44.892288 7860 leader_election.cc:302] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cff622f4d174add86a41f3f68d93654, f8bdd18f33264c138aced249f833e65e; no voters:
I20260925 14:57:44.892908 8079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:44.904512 8079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [term 1 LEADER]: Becoming Leader. State: Replica: 1cff622f4d174add86a41f3f68d93654, State: Running, Role: LEADER
I20260925 14:57:44.905926 8079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } }
I20260925 14:57:44.920051 8086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cff622f4d174add86a41f3f68d93654. Latest consensus state: current_term: 1 leader_uuid: "1cff622f4d174add86a41f3f68d93654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:44.920862 8086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:44.935786 8092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260925 14:57:44.936190 7864 tablet.cc:2369] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:44.950865 8013 tablet.cc:2369] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:44.951828 7984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } attempt: 5
I20260925 14:57:44.957182 8092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:57:44 dist-test-slave-v5vl krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
I20260925 14:57:44.971027 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.952500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:35001 (local address 127.0.0.1:45776)
0925 14:57:44.952837 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.952854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.952949 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.954370 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.954377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.954858 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:44.955068 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.955080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.956776 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.956784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.958269 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.958297 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.959489 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.959495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.959712 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.960423 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:44.960447 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:44.963460 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:44.967855 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.967860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.967864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.968414 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.969032 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:44.969038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:44.969045 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:44.969228 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:44.969955 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:44.969960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:44.970125 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.970331 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.970628 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260925 14:57:44.971359 8100 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:57:44.972225 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.953162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.17.133.252:35001)
0925 14:57:44.953490 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.953499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.953520 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:57:44.953581 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.953587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.953680 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.953829 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.955344 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.956575 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.958469 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.959152 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.959290 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.963680 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.963690 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.963696 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.963734 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.967695 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.968581 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.968587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.968591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.968777 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.969369 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.969377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.969381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.969745 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.969878 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.971557 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.971813 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260925 14:57:44.981920 8034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [term 1 FOLLOWER]: Refusing update from remote peer 1cff622f4d174add86a41f3f68d93654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:44.983009 8079 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:44.987465 7984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e: Bootstrap starting.
I20260925 14:57:45.003324 7984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:45.010210 8065 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cff622f4d174add86a41f3f68d93654. Latest consensus state: current_term: 1 leader_uuid: "1cff622f4d174add86a41f3f68d93654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:45.010567 8065 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:45.014416 8101 mvcc.cc:205] Tried to move back new op lower bound from 7333266493350404096 to 7333266493069639680. Current Snapshot: MvccSnapshot[applied={T|T < 7333266493350404096}]
I20260925 14:57:45.017050 8086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cff622f4d174add86a41f3f68d93654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:45.017336 8086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:45.017748 7984 log.cc:824] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:44.975798 7843 master_runner.cc:186] Error getting master registration for 127.17.133.253:37257: OK, instance_id { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" instance_seqno: 1790348264221796 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:45.019840 8065 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cff622f4d174add86a41f3f68d93654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:45.020097 8065 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:45.024158 7843 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:57:45.032024 8092 catalog_manager.cc:1347] Generated new cluster ID: fb7bef2a8af54f40922702e78fe4bdf5
I20260925 14:57:45.032130 8092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:45.036613 7984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e: No bootstrap required, opened a new log
I20260925 14:57:45.047902 7984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } }
I20260925 14:57:45.048424 7984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037c948e1b6e4878aeb3e3da7f906c2e, State: Initialized, Role: FOLLOWER
I20260925 14:57:45.052577 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.929692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.17.133.253:37257)
0925 14:57:44.929883 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.929888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.949241 (+ 19353us) server_negotiation.cc:407] Connection header received
0925 14:57:44.949487 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.949493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.949575 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.949656 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.012303 (+ 62647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.013360 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.024366 (+ 11006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.025075 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.025175 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.029879 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:45.029886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:45.029895 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:45.029935 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:45.033987 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.049584 (+ 15597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.049590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.049595 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.049775 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.050242 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.050247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.050251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.050609 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:45.050832 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.051966 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.052214 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:57:45.055239 7984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } }
I20260925 14:57:45.062394 8108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:45.064844 8108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:45.064718 7984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:45.065156 7958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:45.068766 7959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cff622f4d174add86a41f3f68d93654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "037c948e1b6e4878aeb3e3da7f906c2e"
I20260925 14:57:45.069877 7958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [term 1 FOLLOWER]: Refusing update from remote peer 1cff622f4d174add86a41f3f68d93654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:45.071066 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.973719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.17.133.252:35001)
0925 14:57:44.997169 (+ 23450us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.997175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.015163 (+ 17988us) server_negotiation.cc:407] Connection header received
0925 14:57:45.016889 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.016902 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.017014 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.017084 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.023763 (+ 6679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.025976 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.028294 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.029529 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.029752 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.054458 (+ 24706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:45.054466 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:45.054470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:45.054524 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:45.060437 (+ 5913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.062825 (+ 2388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.062833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.062836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.062997 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.063445 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.063451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.063454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.063840 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:45.064103 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.070087 (+ 5984us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.070491 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23045}
I20260925 14:57:45.073477 7959 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cff622f4d174add86a41f3f68d93654 in term 1.
I20260925 14:57:45.075646 8086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:45.077726 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.923119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.17.133.254:40045)
0925 14:57:44.923346 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.923355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.945287 (+ 21932us) server_negotiation.cc:407] Connection header received
0925 14:57:44.945841 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.945850 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.945954 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.946040 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.006529 (+ 60489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.007955 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.023987 (+ 16032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.024786 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.024917 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.052887 (+ 27970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:45.052895 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:45.052901 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:45.052948 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:45.058030 (+ 5082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.059566 (+ 1536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.059573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.059577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.059852 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.060797 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.060803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.060838 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.061539 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:45.062229 (+ 690us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.063401 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.063901 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:57:45.126392 8092 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:45.128614 8092 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:45.140568 8092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654: Generated new TSK 0
I20260925 14:57:45.141618 8092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:45.148814 8108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cff622f4d174add86a41f3f68d93654. Latest consensus state: current_term: 1 leader_uuid: "1cff622f4d174add86a41f3f68d93654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:45.149222 8108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:45.149729 8116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cff622f4d174add86a41f3f68d93654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:45.150166 8116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:45.151698 8086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cff622f4d174add86a41f3f68d93654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff622f4d174add86a41f3f68d93654" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 40045 } } peers { permanent_uuid: "037c948e1b6e4878aeb3e3da7f906c2e" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 37257 } } peers { permanent_uuid: "f8bdd18f33264c138aced249f833e65e" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 35001 } } }
I20260925 14:57:45.152050 8086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff622f4d174add86a41f3f68d93654 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:45.204106 8124 catalog_manager.cc:1259] Loaded cluster ID: fb7bef2a8af54f40922702e78fe4bdf5
I20260925 14:57:45.204312 8124 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e: loading cluster ID for follower catalog manager: success
I20260925 14:57:45.208516 8124 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e: acquiring CA information for follower catalog manager: success
I20260925 14:57:45.211342 8124 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 037c948e1b6e4878aeb3e3da7f906c2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 14:57:45.277125 17943 mini_kdc.cc:348] Time spent kdestroy: real 0.112s user 0.001s sys 0.021s
W20260925 14:57:45.337164 7989 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.323526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.17.133.253:37257)
0925 14:57:45.323764 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.323773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.334710 (+ 10937us) server_negotiation.cc:407] Connection header received
0925 14:57:45.335338 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.335349 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.335464 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.335557 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.336909 (+ 1352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48598: BlockingRecv error: recv got EOF from 127.0.0.1:48598 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 14:57:45.339712 7987 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.323386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.17.133.254:40045)
0925 14:57:45.323601 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.323608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.337995 (+ 14387us) server_negotiation.cc:407] Connection header received
0925 14:57:45.338293 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.338300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.338385 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.338458 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.339448 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55964: BlockingRecv error: recv got EOF from 127.0.0.1:55964 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 14:57:45.343678 8062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.341396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.17.133.252:35001)
0925 14:57:45.341577 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.341582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.341940 (+ 358us) server_negotiation.cc:407] Connection header received
0925 14:57:45.342091 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.342097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.342196 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.342291 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.343447 (+ 1156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45784: BlockingRecv error: recv got EOF from 127.0.0.1:45784 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:57:45.392365 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.375801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.17.133.253:37257)
0925 14:57:45.376026 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.376035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.384407 (+ 8372us) server_negotiation.cc:407] Connection header received
0925 14:57:45.384854 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.384862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.385065 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.385213 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.385904 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.387123 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.389331 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.390013 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.390104 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.390870 (+ 766us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.391549 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.391675 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.391918 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.392031 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 14:57:45.434667 8062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.427493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.17.133.252:35001)
0925 14:57:45.427673 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.427680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.432958 (+ 5278us) server_negotiation.cc:407] Connection header received
0925 14:57:45.433207 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.433214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.433354 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.433479 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.434415 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45786: BlockingRecv error: recv got EOF from 127.0.0.1:45786 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:57:45.446280 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.439805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.17.133.253:37257)
0925 14:57:45.440034 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.440041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.440065 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:57:45.440137 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.440142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.440244 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.440347 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.440915 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.442090 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.444086 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.444828 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.444957 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.445442 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.445587 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.445676 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.445849 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.445939 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 14:57:45.456406 8062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.454085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.17.133.252:35001)
0925 14:57:45.454283 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.454289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.454787 (+ 498us) server_negotiation.cc:407] Connection header received
0925 14:57:45.455057 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.455063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.455161 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.455233 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.456179 (+ 946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45788: BlockingRecv error: recv got EOF from 127.0.0.1:45788 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:57:45.469946 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.463021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.17.133.253:37257)
0925 14:57:45.463200 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.463206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.463696 (+ 490us) server_negotiation.cc:407] Connection header received
0925 14:57:45.463872 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.463880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.463974 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.464081 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.464631 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.465848 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.467765 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.468483 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.468644 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.469110 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.469266 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.469359 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.469510 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.469590 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 14:57:45.480466 8062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.478055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.17.133.252:35001)
0925 14:57:45.478280 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.478286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.478753 (+ 467us) server_negotiation.cc:407] Connection header received
0925 14:57:45.479012 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.479019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.479121 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.479207 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.480236 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45798: BlockingRecv error: recv got EOF from 127.0.0.1:45798 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:57:45.495047 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.488525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.17.133.253:37257)
0925 14:57:45.488794 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.488800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.488820 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:57:45.488980 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.488985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.489101 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.489170 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.489790 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.490974 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.492937 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.493666 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.493816 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.494260 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.494395 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.494500 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.494655 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.494737 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
W20260925 14:57:45.503788 8062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.501900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.17.133.252:35001)
0925 14:57:45.502149 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.502166 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.502185 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:57:45.502364 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.502371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.502474 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.502547 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.503572 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45804: BlockingRecv error: recv got EOF from 127.0.0.1:45804 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
I20260925 14:57:45.529780 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.522452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.17.133.253:37257)
0925 14:57:45.522628 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.522634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.523143 (+ 509us) server_negotiation.cc:407] Connection header received
0925 14:57:45.523333 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.523344 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.523451 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.523566 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.524033 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.525266 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.527254 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.528050 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.528192 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.528868 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.529044 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.529111 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.529354 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.529438 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 14:57:45.540316 8062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.537983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.17.133.252:35001)
0925 14:57:45.538209 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.538216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.538769 (+ 553us) server_negotiation.cc:407] Connection header received
0925 14:57:45.538940 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.538949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.539050 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.539169 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.540093 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45820: BlockingRecv error: recv got EOF from 127.0.0.1:45820 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 14:57:45.569701 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.562208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.17.133.253:37257)
0925 14:57:45.562396 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.562404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.562998 (+ 594us) server_negotiation.cc:407] Connection header received
0925 14:57:45.563351 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.563357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.563458 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.563579 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.564064 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.565378 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.567466 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.568175 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.568317 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.568884 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.569029 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.569117 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.569304 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.569396 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 14:57:45.589289 8062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.586800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.17.133.252:35001)
0925 14:57:45.586998 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.587004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.587591 (+ 587us) server_negotiation.cc:407] Connection header received
0925 14:57:45.587803 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.587810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.587919 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.588052 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.589004 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45824: BlockingRecv error: recv got EOF from 127.0.0.1:45824 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:57:45.630609 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.624543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.17.133.253:37257)
0925 14:57:45.624867 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.624874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.625141 (+ 267us) server_negotiation.cc:407] Connection header received
0925 14:57:45.625344 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.625348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.625413 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.625491 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.625996 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.626739 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.628694 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.629268 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.629400 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.629861 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.629967 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.630040 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.630215 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.630277 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
W20260925 14:57:45.639041 8062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.637075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.17.133.252:35001)
0925 14:57:45.637273 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.637279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.637631 (+ 352us) server_negotiation.cc:407] Connection header received
0925 14:57:45.637792 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.637798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.637892 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.638000 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.638753 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45830: BlockingRecv error: recv got EOF from 127.0.0.1:45830 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:57:45.713697 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.706873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.17.133.253:37257)
0925 14:57:45.707049 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.707056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.707608 (+ 552us) server_negotiation.cc:407] Connection header received
0925 14:57:45.707800 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.707806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.707891 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.707987 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.708440 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.709630 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.711557 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.712247 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.712357 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.712880 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.713024 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.713083 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.713303 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.713399 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 14:57:45.730494 8062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:45.724327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.17.133.252:35001)
0925 14:57:45.724539 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.724546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.729159 (+ 4613us) server_negotiation.cc:407] Connection header received
0925 14:57:45.729313 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.729319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.729406 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.729502 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.730300 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45844: BlockingRecv error: recv got EOF from 127.0.0.1:45844 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:57:45.817773 8076 catalog_manager.cc:1259] Loaded cluster ID: fb7bef2a8af54f40922702e78fe4bdf5
I20260925 14:57:45.817873 8076 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e: loading cluster ID for follower catalog manager: success
I20260925 14:57:45.820411 8076 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e: acquiring CA information for follower catalog manager: success
I20260925 14:57:45.822412 8076 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8bdd18f33264c138aced249f833e65e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:57:45.866603 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.861454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.17.133.253:37257)
0925 14:57:45.861710 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.861718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.861993 (+ 275us) server_negotiation.cc:407] Connection header received
0925 14:57:45.862219 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.862223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.862298 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.862400 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.862825 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.863548 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.864921 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.865356 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.865490 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.865885 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.866002 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.866100 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.866255 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.866332 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 14:57:45.877298 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.872067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.17.133.252:35001)
0925 14:57:45.872234 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.872238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.872636 (+ 398us) server_negotiation.cc:407] Connection header received
0925 14:57:45.872822 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.872826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.872974 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.873071 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.873381 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.874068 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.875290 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.875702 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.875790 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.876220 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.876818 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.876934 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.877011 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.877075 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:57:45.891777 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.883994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.17.133.252:35001)
0925 14:57:45.884172 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.884178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.884800 (+ 622us) server_negotiation.cc:407] Connection header received
0925 14:57:45.885040 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.885045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.885149 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.885234 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.886128 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.887179 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.889585 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.890276 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.890440 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.891026 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.891160 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.891272 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.891418 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.891498 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:57:45.892856 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.883450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.17.133.253:37257)
0925 14:57:45.883709 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.883717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.885029 (+ 1312us) server_negotiation.cc:407] Connection header received
0925 14:57:45.885258 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.885264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.885350 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.885438 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.886062 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.887179 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.890489 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.891211 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.891333 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.892078 (+ 745us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.892201 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.892252 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.892487 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.892566 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 14:57:45.892958 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.883238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.17.133.254:40045)
0925 14:57:45.884058 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.884064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.884443 (+ 379us) server_negotiation.cc:407] Connection header received
0925 14:57:45.884634 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.884643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.884903 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.885226 (+ 323us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:45.886322 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.887030 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.890512 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.890971 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.891103 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.891812 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:45.892269 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:45.892396 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.892506 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.892630 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":614,"threads_started":1}
I20260925 14:57:45.897898 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7843
I20260925 14:57:45.899148 7905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:46.008013 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7843
I20260925 14:57:46.028990 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7917
I20260925 14:57:46.030107 7979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:46.136952 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7917
I20260925 14:57:46.156749 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 7992
I20260925 14:57:46.157755 8054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:46.270705 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 7992
2026-09-25T14:57:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4361 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:46 dist-test-slave-v5vl krb5kdc[8237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:46 dist-test-slave-v5vl krb5kdc[8237](info): set up 2 sockets
Sep 25 14:57:46 dist-test-slave-v5vl krb5kdc[8237](info): commencing operation
krb5kdc: starting...
W20260925 14:57:47.365340 17943 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/test-user.keytab.
Sep 25 14:57:47 dist-test-slave-v5vl krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.133.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
Entry for principal kudu/127.17.133.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab.
I20260925 14:57:47.618499 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:43885
--webserver_interface=127.17.133.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:43885,127.17.133.253:45795,127.17.133.252:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:47.947681 8250 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:47.948081 8250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:47.948208 8250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:47.948349 8250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:47.955713 8250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:47.955860 8250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:47.955964 8250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:47.956058 8250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:47.956146 8250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:47.966339 8250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:43885,127.17.133.253:45795,127.17.133.252:39631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.254.keytab
--principal=kudu/127.17.133.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.254:43885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.133.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:47.967705 8250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:47.969043 8250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:47.981186 8258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:47.981238 8256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:47.981204 8255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:47.982041 8250 server_base.cc:1034] running on GCE node
I20260925 14:57:47.982785 8250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:47.984154 8250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:47.985373 8250 hybrid_clock.cc:648] HybridClock initialized: now 1790348267985316 us; error 31 us; skew 500 ppm
I20260925 14:57:47.985709 8250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:47 dist-test-slave-v5vl krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:47.988737 8250 init.cc:377] Logged in from keytab as kudu/127.17.133.254@KRBTEST.COM (short username kudu)
I20260925 14:57:47.990165 8250 webserver.cc:460] Webserver started at http://127.17.133.254:39703/ using document root <none> and password file <none>
I20260925 14:57:47.990880 8250 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:47.991007 8250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:47.991325 8250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:47.994278 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/data/instance:
uuid: "29477d931e554c24a9ebff351f69de9c"
format_stamp: "Formatted at 2026-09-25 14:57:47 on dist-test-slave-v5vl"
server_key: "8bd5b1c8e1a2cc03758fb9c128e8252c"
server_key_iv: "704e1ca2a6270c87a54e5d25f7eab401"
server_key_version: "encryptionkey@0"
I20260925 14:57:47.995059 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/wal/instance:
uuid: "29477d931e554c24a9ebff351f69de9c"
format_stamp: "Formatted at 2026-09-25 14:57:47 on dist-test-slave-v5vl"
server_key: "8bd5b1c8e1a2cc03758fb9c128e8252c"
server_key_iv: "704e1ca2a6270c87a54e5d25f7eab401"
server_key_version: "encryptionkey@0"
I20260925 14:57:48.000211 8250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:57:48.003664 8265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:48.005052 8250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:48.005249 8250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/wal
uuid: "29477d931e554c24a9ebff351f69de9c"
format_stamp: "Formatted at 2026-09-25 14:57:47 on dist-test-slave-v5vl"
server_key: "8bd5b1c8e1a2cc03758fb9c128e8252c"
server_key_iv: "704e1ca2a6270c87a54e5d25f7eab401"
server_key_version: "encryptionkey@0"
I20260925 14:57:48.005472 8250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:48.037364 8250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:48.041505 8250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:48.041790 8250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:48.060468 8250 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.254:43885
I20260925 14:57:48.060480 8316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.254:43885 every 8 connection(s)
I20260925 14:57:48.061937 8250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:48.063506 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 8250
I20260925 14:57:48.063678 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-0/wal/instance
I20260925 14:57:48.064193 17943 external_mini_cluster.cc:1428] Setting key a1ff9be2cb88e6295fa593eb02c20f06
I20260925 14:57:48.066656 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:48.076278 8317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.133.253@KRBTEST.COM; defaulting to no policy
Sep 25 14:57:48 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.253@KRBTEST.COM" created.
I20260925 14:57:48.099498 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.079636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:43885 (local address 127.0.0.1:33280)
0925 14:57:48.080757 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.080798 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:48.082513 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:48.083341 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:48.083349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:48.084075 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:48.084413 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:48.084431 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.086018 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.086025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:48.087212 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:48.087224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.088134 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.088141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:48.088316 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.089021 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:48.089042 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:48.091375 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:48.095442 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:48.095452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:48.095461 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:48.095869 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:48.096333 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:48.096336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:48.096339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:48.096486 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:48.097492 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:48.097499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:48.097717 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 14:57:48.097963 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 14:57:48.098381 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"spinlock_wait_cycles":15232,"thread_start_us":687,"threads_started":1}
I20260925 14:57:48.099534 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.079627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.17.133.254:43885)
0925 14:57:48.080450 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:48.080467 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:48.080913 (+ 446us) server_negotiation.cc:407] Connection header received
0925 14:57:48.082879 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:48.082884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:48.083126 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:48.083341 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:48.084670 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.085859 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.087364 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.087905 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.088101 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.091579 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:48.091586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:48.091598 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:48.091626 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:48.095296 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.096008 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.096019 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.096025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.096215 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.096620 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.096628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.096633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.097333 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:48.097539 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:48.098145 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 14:57:48.098584 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":575,"thread_start_us":368,"threads_started":1}
I20260925 14:57:48.102564 8317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:48.103804 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.103376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32774)
0925 14:57:48.103551 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.103666 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 14:57:48.104110 8268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
I20260925 14:57:48.105826 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.105379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32776)
0925 14:57:48.105554 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.105652 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 14:57:48.106053 8317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:45795: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab.
Entry for principal kudu/127.17.133.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab.
I20260925 14:57:48.125468 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.253:45795
--webserver_interface=127.17.133.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab
--principal=kudu/127.17.133.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:43885,127.17.133.253:45795,127.17.133.252:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:48.155431 8317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } attempt: 1
I20260925 14:57:48.156764 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.156161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32786)
0925 14:57:48.156384 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.156551 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 14:57:48.157796 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.157406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32798)
0925 14:57:48.157548 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.157630 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 14:57:48.157984 8317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:45795: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
I20260925 14:57:48.226266 8317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } attempt: 2
I20260925 14:57:48.227659 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.227035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32808)
0925 14:57:48.227313 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.227441 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 14:57:48.228709 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.228312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32812)
0925 14:57:48.228474 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.228550 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 14:57:48.228905 8317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:45795: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
I20260925 14:57:48.320182 8317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } attempt: 3
I20260925 14:57:48.321638 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.321014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32828)
0925 14:57:48.321294 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.321428 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 14:57:48.322845 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.322352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32834)
0925 14:57:48.322540 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.322644 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 14:57:48.323113 8317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:45795: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
W20260925 14:57:48.460647 8324 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:48.460954 8324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:48.461024 8324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:48.461109 8324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 14:57:48.466400 8317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } attempt: 4
I20260925 14:57:48.467619 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.467039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32846)
0925 14:57:48.467361 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.467460 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260925 14:57:48.468606 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.468199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32856)
0925 14:57:48.468346 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.468448 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 14:57:48.468814 8324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:48.468823 8317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.253:45795: Network error: Client connection negotiation failed: client connection to 127.17.133.253:45795: connect: Connection refused (error 111)
W20260925 14:57:48.468926 8324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:48.469017 8324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:48.469106 8324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:48.469195 8324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:48.479000 8324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:43885,127.17.133.253:45795,127.17.133.252:39631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.253.keytab
--principal=kudu/127.17.133.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.253:45795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.133.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:48.480314 8324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:48.481691 8324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:48.493112 8330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:48.493031 8332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:48.493144 8329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:48.494302 8324 server_base.cc:1034] running on GCE node
I20260925 14:57:48.494971 8324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:48.496245 8324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:48.497483 8324 hybrid_clock.cc:648] HybridClock initialized: now 1790348268497429 us; error 60 us; skew 500 ppm
I20260925 14:57:48.497792 8324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:48 dist-test-slave-v5vl krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:48.500959 8324 init.cc:377] Logged in from keytab as kudu/127.17.133.253@KRBTEST.COM (short username kudu)
I20260925 14:57:48.502379 8324 webserver.cc:460] Webserver started at http://127.17.133.253:40053/ using document root <none> and password file <none>
I20260925 14:57:48.503067 8324 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:48.503196 8324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:48.503507 8324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:48.506342 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/data/instance:
uuid: "96b5712655ae4cd5a36bbf36aa687570"
format_stamp: "Formatted at 2026-09-25 14:57:48 on dist-test-slave-v5vl"
server_key: "5fd94fd4bf8f01c657259e7801f0a830"
server_key_iv: "f5400b9a434ff00a285297d0d92b9083"
server_key_version: "encryptionkey@0"
I20260925 14:57:48.507112 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/wal/instance:
uuid: "96b5712655ae4cd5a36bbf36aa687570"
format_stamp: "Formatted at 2026-09-25 14:57:48 on dist-test-slave-v5vl"
server_key: "5fd94fd4bf8f01c657259e7801f0a830"
server_key_iv: "f5400b9a434ff00a285297d0d92b9083"
server_key_version: "encryptionkey@0"
I20260925 14:57:48.512303 8324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 14:57:48.515810 8339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:48.517257 8324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:48.517439 8324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/wal
uuid: "96b5712655ae4cd5a36bbf36aa687570"
format_stamp: "Formatted at 2026-09-25 14:57:48 on dist-test-slave-v5vl"
server_key: "5fd94fd4bf8f01c657259e7801f0a830"
server_key_iv: "f5400b9a434ff00a285297d0d92b9083"
server_key_version: "encryptionkey@0"
I20260925 14:57:48.517656 8324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:48.535965 8324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:48.540362 8324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:48.540668 8324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:48.559072 8324 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.253:45795
I20260925 14:57:48.559082 8390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.253:45795 every 8 connection(s)
I20260925 14:57:48.560621 8324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/data/info.pb
I20260925 14:57:48.564155 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:48.570345 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 8324
I20260925 14:57:48.570525 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-1/wal/instance
I20260925 14:57:48.570950 17943 external_mini_cluster.cc:1428] Setting key 75f365fe95a52bec7d0fb4522bda821a
I20260925 14:57:48.570457 8391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:48 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:48.589763 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.573138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.17.133.254:43885)
0925 14:57:48.573312 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:48.573323 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:48.574561 (+ 1238us) server_negotiation.cc:407] Connection header received
0925 14:57:48.576423 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:48.576429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:48.576516 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:48.576642 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:48.578096 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.578797 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.580280 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.580780 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.580915 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.584199 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:48.584204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:48.584208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:48.584239 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:48.586830 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.587554 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.587566 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.587568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.587681 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.588093 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.588098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.588102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.588380 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:48.588535 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:48.589210 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 14:57:48.589447 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:57:48.590811 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.573688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:43885 (local address 127.0.0.1:33282)
0925 14:57:48.574399 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.574441 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:48.576154 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:48.576874 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:48.576883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:48.577386 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:48.577924 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:48.577941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.578960 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.578964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:48.580117 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:48.580128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.580991 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.580998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:48.581230 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.581910 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:48.581933 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:48.583987 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:48.586971 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:48.586991 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:48.587008 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:48.587432 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:48.587806 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:48.587811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:48.587814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:48.587986 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:48.588518 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:48.588525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:48.588870 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 14:57:48.589128 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 14:57:48.589633 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":300,"threads_started":1}
I20260925 14:57:48.592926 8391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.133.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.133.252@KRBTEST.COM" created.
Sep 25 14:57:48 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
I20260925 14:57:48.609826 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.593977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32866)
0925 14:57:48.594135 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.594152 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:48.594240 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:48.595872 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:48.595876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:48.596220 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:48.596362 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:48.596370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.598264 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.598268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:48.599089 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:48.599100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.600281 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.600286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:48.600474 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.601124 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:48.601142 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:48.603534 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:48.607285 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:48.607289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:48.607292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:48.607625 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:48.608014 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:48.608018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:48.608022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:48.608212 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:48.608833 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:48.608842 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:48.608982 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 14:57:48.609138 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 14:57:48.609417 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":37}
I20260925 14:57:48.609917 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.594605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.17.133.253:45795)
0925 14:57:48.595355 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:48.595362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:48.595391 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:57:48.595437 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:48.595442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:48.595710 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:48.595865 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:48.596635 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.598095 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.599226 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.600147 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.600283 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.603743 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:48.603755 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:48.603769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:48.603796 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:48.607158 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.607765 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.607774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.607785 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.607900 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.608296 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.608304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.608307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.608709 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:48.608916 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:48.609315 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 14:57:48.609530 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":546,"thread_start_us":329,"threads_started":1}
I20260925 14:57:48.612706 8391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:48.613813 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.613363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50142)
0925 14:57:48.613549 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.613658 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 14:57:48.614037 8342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
I20260925 14:57:48.615792 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.615426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50156)
0925 14:57:48.615587 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.615687 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 14:57:48.616034 8391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:39631: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.133.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab.
Entry for principal kudu/127.17.133.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab.
I20260925 14:57:48.631707 17943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
/tmp/dist-test-taskNtxnNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.252:39631
--webserver_interface=127.17.133.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab
--principal=kudu/127.17.133.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.133.212:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.133.254:43885,127.17.133.253:45795,127.17.133.252:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:48.665365 8391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } attempt: 1
I20260925 14:57:48.666563 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.666020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50162)
0925 14:57:48.666252 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.666380 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 14:57:48.667641 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.667255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50168)
0925 14:57:48.667419 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.667511 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 14:57:48.667924 8391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:39631: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
I20260925 14:57:48.736172 8391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } attempt: 2
I20260925 14:57:48.737428 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.736881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50180)
0925 14:57:48.737126 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.737239 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 14:57:48.738525 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.738143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50192)
0925 14:57:48.738298 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.738379 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 14:57:48.738745 8391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:39631: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
I20260925 14:57:48.768077 8317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } attempt: 5
Sep 25 14:57:48 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
I20260925 14:57:48.785338 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.769153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.17.133.253:45795)
0925 14:57:48.769365 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:48.769371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:48.769395 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:57:48.769462 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:48.769466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:48.769524 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:48.769654 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:48.770398 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.771116 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.772309 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.772822 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.772964 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.780067 (+ 7103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:48.780072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:48.780075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:48.780108 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:48.782746 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.783425 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.783437 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.783439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.783556 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.783934 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.783940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.783943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.784228 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:48.784384 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:48.784797 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 14:57:48.785022 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:57:48.785353 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.768890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32882)
0925 14:57:48.769083 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.769103 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:48.769244 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:48.769697 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:48.769703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:48.770095 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:48.770252 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:48.770260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.771277 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.771283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:48.772161 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:48.772172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.772969 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.772975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:48.773176 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.777745 (+ 4569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:48.777765 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:48.779846 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:48.782907 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:48.782912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:48.782916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:48.783285 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:48.783669 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:48.783674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:48.783677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:48.783826 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:48.784354 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:48.784359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:48.784513 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 14:57:48.784694 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 14:57:48.785000 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 14:57:48.786638 8317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:48.787582 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.787216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50200)
0925 14:57:48.787379 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.787458 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 14:57:48.788456 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.788132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50210)
0925 14:57:48.788294 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.788355 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 14:57:48.788697 8317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:39631: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
I20260925 14:57:48.829998 8391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } attempt: 3
I20260925 14:57:48.831276 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.830698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50216)
0925 14:57:48.830958 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.831079 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 14:57:48.832418 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.831917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50222)
0925 14:57:48.832178 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.832285 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260925 14:57:48.832651 8391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:39631: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
I20260925 14:57:48.839910 8317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } attempt: 1
I20260925 14:57:48.841097 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.840484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50236)
0925 14:57:48.840754 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.840884 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 14:57:48.842108 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.841792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50240)
0925 14:57:48.841930 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.841999 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 14:57:48.842336 8317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:39631: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
I20260925 14:57:48.910600 8317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } attempt: 2
I20260925 14:57:48.911794 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.911307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50248)
0925 14:57:48.911506 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.911630 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 14:57:48.912863 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.912483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50252)
0925 14:57:48.912669 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.912744 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 14:57:48.913091 8317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:39631: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
W20260925 14:57:48.962880 8398 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:48.963191 8398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:48.963335 8398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:48.963469 8398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:48.970556 8398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:48.970635 8398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:48.970748 8398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:48.970834 8398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:48.970942 8398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:48.975881 8391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } attempt: 4
I20260925 14:57:48.977029 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.976527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50260)
0925 14:57:48.976783 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.976877 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 14:57:48.978075 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.977753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50262)
0925 14:57:48.977903 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.977974 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 14:57:48.978323 8391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:39631: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
I20260925 14:57:48.981496 8398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.133.212:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.133.254:43885,127.17.133.253:45795,127.17.133.252:39631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/krb5kdc/kudu_127.17.133.252.keytab
--principal=kudu/127.17.133.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.133.252:39631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.133.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:57:48.982761 8398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:48.984061 8398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:48.995520 8403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:48.995450 8404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:48.995496 8406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:48.996619 8398 server_base.cc:1034] running on GCE node
I20260925 14:57:48.997433 8398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:48.998862 8398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:49.000118 8398 hybrid_clock.cc:648] HybridClock initialized: now 1790348269000048 us; error 44 us; skew 500 ppm
I20260925 14:57:49.000464 8398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:57:48 dist-test-slave-v5vl krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:49.003666 8398 init.cc:377] Logged in from keytab as kudu/127.17.133.252@KRBTEST.COM (short username kudu)
I20260925 14:57:49.005146 8398 webserver.cc:460] Webserver started at http://127.17.133.252:35501/ using document root <none> and password file <none>
I20260925 14:57:49.005867 8398 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:49.005996 8398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:49.006309 8398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:49.009325 8398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/data/instance:
uuid: "f303346c6ab244749d73766002f902d8"
format_stamp: "Formatted at 2026-09-25 14:57:49 on dist-test-slave-v5vl"
server_key: "0902470bc5a130c80e11f70ee19a20cb"
server_key_iv: "76721cd68ef859cd2c5c2560ab57a394"
server_key_version: "encryptionkey@0"
I20260925 14:57:49.010123 8398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/wal/instance:
uuid: "f303346c6ab244749d73766002f902d8"
format_stamp: "Formatted at 2026-09-25 14:57:49 on dist-test-slave-v5vl"
server_key: "0902470bc5a130c80e11f70ee19a20cb"
server_key_iv: "76721cd68ef859cd2c5c2560ab57a394"
server_key_version: "encryptionkey@0"
I20260925 14:57:49.015331 8398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:57:49.018743 8413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:49.019330 8317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } attempt: 3
I20260925 14:57:49.020172 8398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:57:49.020516 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.019921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50276)
0925 14:57:49.020164 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.020276 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 14:57:49.020408 8398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/wal
uuid: "f303346c6ab244749d73766002f902d8"
format_stamp: "Formatted at 2026-09-25 14:57:49 on dist-test-slave-v5vl"
server_key: "0902470bc5a130c80e11f70ee19a20cb"
server_key_iv: "76721cd68ef859cd2c5c2560ab57a394"
server_key_version: "encryptionkey@0"
I20260925 14:57:49.020696 8398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:49.021814 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.021334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50290)
0925 14:57:49.021559 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.021684 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260925 14:57:49.022104 8317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.133.252:39631: Network error: Client connection negotiation failed: client connection to 127.17.133.252:39631: connect: Connection refused (error 111)
I20260925 14:57:49.041481 8398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:49.045900 8398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:49.046190 8398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:49.064625 8398 rpc_server.cc:307] RPC server started. Bound to: 127.17.133.252:39631
I20260925 14:57:49.064651 8464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.133.252:39631 every 8 connection(s)
I20260925 14:57:49.065977 8398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/data/info.pb
I20260925 14:57:49.066671 17943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu as pid 8398
I20260925 14:57:49.066877 17943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtxnNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347918228911-17943-0/minicluster-data/master-2/wal/instance
I20260925 14:57:49.067571 17943 external_mini_cluster.cc:1428] Setting key 23286d21ef8b1ae2243bdd24cbb00ae1
I20260925 14:57:49.070570 8465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:49.079463 8465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:49 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
Sep 25 14:57:49 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
I20260925 14:57:49.091710 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.070996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.17.133.252:39631)
0925 14:57:49.071761 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.071772 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.071811 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:57:49.073420 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.073440 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.073825 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.074159 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.075859 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.077234 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.078983 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.079631 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.079764 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.084512 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.084520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.084529 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.084570 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.087510 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.088188 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.088198 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.088208 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.088406 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.088889 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.088892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.088894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.089381 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.089642 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.090135 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.090554 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":348,"threads_started":1}
I20260925 14:57:49.097914 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.082481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.17.133.254:43885)
0925 14:57:49.082725 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.082732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.083298 (+ 566us) server_negotiation.cc:407] Connection header received
0925 14:57:49.083506 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.083513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.083596 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.083703 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.084835 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.085541 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.087054 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.087482 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.087621 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.091249 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.091254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.091257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.091290 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.095210 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.095874 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.095881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.095885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.096017 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.096374 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.096380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.096383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.096726 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.096911 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.097331 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.097539 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:57:49.097944 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.082469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.254:43885 (local address 127.0.0.1:33288)
0925 14:57:49.083167 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.083204 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:49.083382 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:49.083756 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:49.083763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:49.084359 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:49.084689 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.084702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.085694 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.085700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.086923 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.086934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.087664 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.087670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.087849 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.088439 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:49.088455 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:49.091048 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:49.095334 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.095346 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.095355 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.095751 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.096124 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.096130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.096134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.096263 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.096857 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:49.096867 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:49.097056 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 14:57:49.097237 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 14:57:49.097519 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":389,"threads_started":1}
I20260925 14:57:49.099306 8465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:49 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.254@KRBTEST.COM
Sep 25 14:57:49 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
I20260925 14:57:49.110322 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.094690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.17.133.254:43885)
0925 14:57:49.095292 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.095297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.095315 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:57:49.095354 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.095358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.095408 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.095524 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.096729 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.097560 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.099351 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.099948 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.100076 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.104754 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.104758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.104761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.104789 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.107285 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.108040 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.108046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.108050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.108176 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.108631 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.108636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.108639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.109030 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.109352 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.109601 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.109829 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":266,"threads_started":1}
I20260925 14:57:49.113729 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.100141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.17.133.253:45795)
0925 14:57:49.100325 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.100330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.100351 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:57:49.100506 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.100514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.100620 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.100710 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.101407 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.102534 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.104060 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.104455 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.104544 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.107663 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.107667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.107670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.107718 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.110886 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.111636 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.111640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.111643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.111803 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.112272 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.112279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.112289 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.112755 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.112909 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.113299 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.113482 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 14:57:49.113832 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.100058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.253:45795 (local address 127.0.0.1:32892)
0925 14:57:49.100300 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.100314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:49.100383 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:49.100771 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:49.100778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:49.101126 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:49.101265 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.101272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.102668 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.102671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.103927 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.103943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.104652 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.104660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.104832 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.105376 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:49.105402 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:49.107474 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:49.110998 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.111002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.111005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.111527 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.111930 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.111938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.111943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.112112 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.112874 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:49.112878 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:49.113005 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 14:57:49.113158 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 14:57:49.113444 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260925 14:57:49.114873 8465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:49 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.133.253@KRBTEST.COM
Sep 25 14:57:49 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.252@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
I20260925 14:57:49.128314 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.112152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.17.133.253:45795)
0925 14:57:49.112884 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.112889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.112911 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:57:49.112962 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.112965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.113013 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.113089 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.114638 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.115555 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.117204 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.117583 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.117668 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.122621 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.122628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.122633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.122666 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.125600 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.126304 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.126311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.126315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.126467 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.126854 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.126860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.126867 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.127170 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.127433 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.127693 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.127895 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":356,"threads_started":1}
I20260925 14:57:49.128805 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.115526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50314)
0925 14:57:49.115662 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.115697 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:49.115783 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:49.116210 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:49.116212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:49.116543 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:49.116734 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.116742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.117798 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.117803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.118637 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.118647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.119619 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.119623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.119780 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.120288 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:49.120303 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:49.122695 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:49.126361 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.126369 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.126372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.126777 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.127140 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.127144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.127149 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.127301 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.127871 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:49.127876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:49.128013 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 14:57:49.128163 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 14:57:49.128460 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:57:49.129334 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.115749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.17.133.252:39631)
0925 14:57:49.115931 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.115937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.115954 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:57:49.116015 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.116020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.116087 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.116186 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.116864 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.117662 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.118781 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.119479 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.119613 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.122863 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.122867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.122870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.122896 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.125603 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.126908 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.126912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.126914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.127028 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.127421 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.127426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.127430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.127746 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.127894 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.128315 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.128996 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":396,"server-negotiator.queue_time_us":58}
I20260925 14:57:49.133052 17943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:49.134642 8465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8: Bootstrap starting.
I20260925 14:57:49.137822 8465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:49.135824 8292 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "29477d931e554c24a9ebff351f69de9c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33292
I20260925 14:57:49.138912 8465 log.cc:824] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:49.141535 8465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8: No bootstrap required, opened a new log
I20260925 14:57:49.148236 8465 raft_consensus.cc:348] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } }
I20260925 14:57:49.148559 8465 raft_consensus.cc:729] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f303346c6ab244749d73766002f902d8, State: Initialized, Role: FOLLOWER
I20260925 14:57:49.149027 8465 consensus_queue.cc:260] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } }
I20260925 14:57:49.149962 8473 sys_catalog.cc:455] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.150373 8473 sys_catalog.cc:458] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:49.151038 8465 sys_catalog.cc:564] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 14:57:49.160444 8484 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:49.160626 8484 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:49.158470 8398 master_runner.cc:186] Error getting master registration for 127.17.133.254:43885: OK, instance_id { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" instance_seqno: 1790348268054982 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:49.168390 8398 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:57:49.199368 8317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } attempt: 4
Sep 25 14:57:49 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.254@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
I20260925 14:57:49.216713 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.200216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.17.133.252:39631)
0925 14:57:49.200475 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.200481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.200507 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:57:49.200604 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.200610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.200691 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.200820 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.201782 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.202961 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.204870 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.205579 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.205716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.209899 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.209908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.209915 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.209952 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.213519 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.214359 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.214365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.214368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.214512 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.214983 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.214988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.214991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.215447 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.215621 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.216086 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.216307 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260925 14:57:49.216706 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.200140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50320)
0925 14:57:49.200363 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.200381 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:49.200487 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:49.200857 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:49.200862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:49.201390 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:49.201607 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.201622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.203092 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.203097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.204720 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.204737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.205728 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.205733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.205921 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.206636 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:49.206659 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:49.209640 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:49.213681 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.213688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.213693 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.214234 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.214639 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.214645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.214649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.214865 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.215579 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:49.215585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:49.215798 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 14:57:49.215964 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 14:57:49.216268 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 14:57:49.222815 8317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c: Bootstrap starting.
I20260925 14:57:49.226863 8317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:49.228127 8317 log.cc:824] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:49.231701 8317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c: No bootstrap required, opened a new log
I20260925 14:57:49.239470 8317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } }
I20260925 14:57:49.239918 8317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29477d931e554c24a9ebff351f69de9c, State: Initialized, Role: FOLLOWER
I20260925 14:57:49.240434 8317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } }
I20260925 14:57:49.241465 8487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.241840 8487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:49.242753 8317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:49.242777 8292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:57:49.243026 8292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:49.245641 8292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } }
I20260925 14:57:49.246948 8292 leader_election.cc:288] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [CANDIDATE]: Term 1 election: Requested vote from peers 96b5712655ae4cd5a36bbf36aa687570 (127.17.133.253:45795), f303346c6ab244749d73766002f902d8 (127.17.133.252:39631)
I20260925 14:57:49.247978 8440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29477d931e554c24a9ebff351f69de9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f303346c6ab244749d73766002f902d8"
I20260925 14:57:49.248521 8440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:49.253365 8440 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29477d931e554c24a9ebff351f69de9c in term 1.
I20260925 14:57:49.254205 8268 leader_election.cc:302] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29477d931e554c24a9ebff351f69de9c, f303346c6ab244749d73766002f902d8; no voters:
I20260925 14:57:49.254791 8487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:49.258060 8487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [term 1 LEADER]: Becoming Leader. State: Replica: 29477d931e554c24a9ebff351f69de9c, State: Running, Role: LEADER
I20260925 14:57:49.258905 8487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } }
I20260925 14:57:49.274974 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29477d931e554c24a9ebff351f69de9c. Latest consensus state: current_term: 1 leader_uuid: "29477d931e554c24a9ebff351f69de9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.275398 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:49.276348 8499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:49.277547 8391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } attempt: 5
Sep 25 14:57:49 dist-test-slave-v5vl krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.133.253@KRBTEST.COM for kudu/127.17.133.252@KRBTEST.COM
W20260925 14:57:49.294554 8271 tablet.cc:2369] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:49.298187 8419 tablet.cc:2369] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:49.300239 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.278366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.133.252:39631 (local address 127.0.0.1:50336)
0925 14:57:49.278861 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.278888 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:49.279037 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:49.280149 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:49.280157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:49.280855 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:49.281133 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.281149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.282791 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.282796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.284328 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:49.284348 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.286112 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.286122 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:49.286424 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.287472 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:49.287507 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:49.291501 (+ 3994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:49.296922 (+ 5421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.296930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.296937 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.297448 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.297924 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:49.297928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:49.297932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:49.298170 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:49.298954 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:49.298962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:49.299178 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 14:57:49.299371 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 14:57:49.299746 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331}
I20260925 14:57:49.301575 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.279350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.17.133.252:39631)
0925 14:57:49.279570 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.279579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.279602 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:57:49.279682 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.279690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.279788 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.279905 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.281377 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.282614 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.284767 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.285707 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.285852 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.291756 (+ 5904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.291767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.291775 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.291828 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.296707 (+ 4879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.297599 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.297607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.297613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.297779 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.298393 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.298402 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.298405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.298740 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.298885 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.300754 (+ 1869us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.301060 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:57:49.310149 8391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570: Bootstrap starting.
I20260925 14:57:49.315302 8391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:49.319191 8391 log.cc:824] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:49.335896 8391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570: No bootstrap required, opened a new log
I20260925 14:57:49.347046 8391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } }
I20260925 14:57:49.347461 8391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b5712655ae4cd5a36bbf36aa687570, State: Initialized, Role: FOLLOWER
I20260925 14:57:49.347986 8391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } }
I20260925 14:57:49.348610 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.283911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.17.133.254:43885)
0925 14:57:49.284684 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.284693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.307593 (+ 22900us) server_negotiation.cc:407] Connection header received
0925 14:57:49.308035 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.308041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.315777 (+ 7736us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.315916 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.318922 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.320538 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.323291 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.324305 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.324436 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.332983 (+ 8547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.332993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.333000 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.333055 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.337399 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.342837 (+ 5438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.342843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.342847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.343026 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.346635 (+ 3609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.346651 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.346654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.347016 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.347227 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.347891 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.348184 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594}
I20260925 14:57:49.353605 8499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:49.357417 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.329785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.17.133.252:39631)
0925 14:57:49.329953 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.329958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.334977 (+ 5019us) server_negotiation.cc:407] Connection header received
0925 14:57:49.335294 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.335301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.335427 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.335559 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.337373 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.338458 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.340282 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.340984 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.341089 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.344367 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.344374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.344379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.344416 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.348021 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.349115 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.349121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.349124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.349282 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.349714 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.349721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.349727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.350026 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.350277 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.356779 (+ 6502us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.357016 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:57:49.362987 8510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.363539 8510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:49.364380 8391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:49.364403 8366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29477d931e554c24a9ebff351f69de9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "96b5712655ae4cd5a36bbf36aa687570"
I20260925 14:57:49.364887 8366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:49.378769 8366 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29477d931e554c24a9ebff351f69de9c in term 1.
I20260925 14:57:49.366928 8250 master_runner.cc:186] Error getting master registration for 127.17.133.253:45795: OK, instance_id { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" instance_seqno: 1790348268553947 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:49.392742 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.289952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.17.133.253:45795)
0925 14:57:49.290140 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.290145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.329176 (+ 39031us) server_negotiation.cc:407] Connection header received
0925 14:57:49.330505 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.330514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.330614 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.330711 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.343551 (+ 12840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.344797 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.350555 (+ 5758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:49.351170 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:49.351263 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:49.361828 (+ 10565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:49.361835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:49.361840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:49.361886 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:49.379082 (+ 17196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.388737 (+ 9655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.388748 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.388755 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.388994 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:49.389634 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:49.389654 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:49.389659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:49.390152 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:49.391013 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:49.391798 (+ 785us) server_negotiation.cc:299] Negotiation successful
0925 14:57:49.392190 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:57:49.396715 8250 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:57:49.398488 8366 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [term 1 FOLLOWER]: Refusing update from remote peer 29477d931e554c24a9ebff351f69de9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:49.398536 8440 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [term 1 FOLLOWER]: Refusing update from remote peer 29477d931e554c24a9ebff351f69de9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:49.399597 8487 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:49.400283 8487 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:49.429874 8510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29477d931e554c24a9ebff351f69de9c. Latest consensus state: current_term: 1 leader_uuid: "29477d931e554c24a9ebff351f69de9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.430245 8510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:49.434376 8518 mvcc.cc:205] Tried to move back new op lower bound from 7333266511445291008 to 7333266510896234496. Current Snapshot: MvccSnapshot[applied={T|T < 7333266511445291008}]
I20260925 14:57:49.436741 8512 mvcc.cc:205] Tried to move back new op lower bound from 7333266511445291008 to 7333266510896234496. Current Snapshot: MvccSnapshot[applied={T|T < 7333266511445291008}]
I20260925 14:57:49.443758 8487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29477d931e554c24a9ebff351f69de9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.444144 8487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:49.449179 8510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29477d931e554c24a9ebff351f69de9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.449508 8510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:49.453437 8499 catalog_manager.cc:1347] Generated new cluster ID: b48c6cd297244956ab589e85cd8e27cc
I20260925 14:57:49.453552 8499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:49.457858 8473 sys_catalog.cc:455] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29477d931e554c24a9ebff351f69de9c. Latest consensus state: current_term: 1 leader_uuid: "29477d931e554c24a9ebff351f69de9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.458182 8473 sys_catalog.cc:458] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:49.460427 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29477d931e554c24a9ebff351f69de9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.460681 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:49.463531 8473 sys_catalog.cc:455] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29477d931e554c24a9ebff351f69de9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29477d931e554c24a9ebff351f69de9c" member_type: VOTER last_known_addr { host: "127.17.133.254" port: 43885 } } peers { permanent_uuid: "96b5712655ae4cd5a36bbf36aa687570" member_type: VOTER last_known_addr { host: "127.17.133.253" port: 45795 } } peers { permanent_uuid: "f303346c6ab244749d73766002f902d8" member_type: VOTER last_known_addr { host: "127.17.133.252" port: 39631 } } }
I20260925 14:57:49.463759 8473 sys_catalog.cc:458] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:49.463819 8529 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:57:49.477787 8499 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:49.479667 8499 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:49.494735 8499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29477d931e554c24a9ebff351f69de9c: Generated new TSK 0
I20260925 14:57:49.495785 8499 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:57:49.606071 8470 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.591409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.17.133.254:43885)
0925 14:57:49.591624 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.591631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.603590 (+ 11959us) server_negotiation.cc:407] Connection header received
0925 14:57:49.604168 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.604176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.604272 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.604359 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.605764 (+ 1405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33312: BlockingRecv error: recv got EOF from 127.0.0.1:33312 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 14:57:49.608392 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.590979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.17.133.253:45795)
0925 14:57:49.591427 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.591436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.606178 (+ 14742us) server_negotiation.cc:407] Connection header received
0925 14:57:49.606436 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.606444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.606554 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.606628 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.608054 (+ 1426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32912: BlockingRecv error: recv got EOF from 127.0.0.1:32912 (error 108)
Metrics: {"server-negotiator.queue_time_us":298}
W20260925 14:57:49.608711 8468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.606405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.17.133.252:39631)
0925 14:57:49.606604 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.606610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.607057 (+ 447us) server_negotiation.cc:407] Connection header received
0925 14:57:49.607317 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.607323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.607419 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.607497 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.608430 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50356: BlockingRecv error: recv got EOF from 127.0.0.1:50356 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 14:57:49.622292 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.614777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.17.133.253:45795)
0925 14:57:49.615006 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.615016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.620794 (+ 5778us) server_negotiation.cc:407] Connection header received
0925 14:57:49.620968 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.620974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.621099 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.621234 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.622069 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32920: BlockingRecv error: recv got EOF from 127.0.0.1:32920 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 14:57:49.628566 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.626581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.17.133.253:45795)
0925 14:57:49.626759 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.626764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.627230 (+ 466us) server_negotiation.cc:407] Connection header received
0925 14:57:49.627414 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.627420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.627498 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.627592 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.628336 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32922: BlockingRecv error: recv got EOF from 127.0.0.1:32922 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 14:57:49.635874 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.633801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.17.133.253:45795)
0925 14:57:49.633970 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.633975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.634477 (+ 502us) server_negotiation.cc:407] Connection header received
0925 14:57:49.634657 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.634663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.634746 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.634850 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.635614 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32934: BlockingRecv error: recv got EOF from 127.0.0.1:32934 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 14:57:49.646049 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.643900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.17.133.253:45795)
0925 14:57:49.644071 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.644077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.644548 (+ 471us) server_negotiation.cc:407] Connection header received
0925 14:57:49.644772 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.644780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.644881 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.645002 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.645808 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32950: BlockingRecv error: recv got EOF from 127.0.0.1:32950 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 14:57:49.659734 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.657587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.17.133.253:45795)
0925 14:57:49.657771 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.657777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.658302 (+ 525us) server_negotiation.cc:407] Connection header received
0925 14:57:49.658476 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.658482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.658591 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.658720 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.659506 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32954: BlockingRecv error: recv got EOF from 127.0.0.1:32954 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 14:57:49.681459 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.679207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.17.133.253:45795)
0925 14:57:49.679400 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.679407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.679988 (+ 581us) server_negotiation.cc:407] Connection header received
0925 14:57:49.680169 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.680175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.680269 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.680373 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.681195 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32956: BlockingRecv error: recv got EOF from 127.0.0.1:32956 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 14:57:49.719275 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.717142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.17.133.253:45795)
0925 14:57:49.717437 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.717445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.717855 (+ 410us) server_negotiation.cc:407] Connection header received
0925 14:57:49.718052 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.718056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.718140 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.718244 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.719043 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32958: BlockingRecv error: recv got EOF from 127.0.0.1:32958 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260925 14:57:49.789067 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.786979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.17.133.253:45795)
0925 14:57:49.787217 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.787224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.787652 (+ 428us) server_negotiation.cc:407] Connection header received
0925 14:57:49.787816 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.787823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.787941 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.788086 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.788827 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32974: BlockingRecv error: recv got EOF from 127.0.0.1:32974 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 14:57:49.924018 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:49.921970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.17.133.253:45795)
0925 14:57:49.922220 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:49.922229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:49.922527 (+ 298us) server_negotiation.cc:407] Connection header received
0925 14:57:49.922772 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:49.922778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:49.922867 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:49.922979 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:49.923766 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32988: BlockingRecv error: recv got EOF from 127.0.0.1:32988 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:57:50.162576 8484 catalog_manager.cc:1259] Loaded cluster ID: b48c6cd297244956ab589e85cd8e27cc
I20260925 14:57:50.162672 8484 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8: loading cluster ID for follower catalog manager: success
I20260925 14:57:50.165409 8484 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8: acquiring CA information for follower catalog manager: success
I20260925 14:57:50.167137 8484 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f303346c6ab244749d73766002f902d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 14:57:50.184844 8471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:50.183198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.17.133.253:45795)
0925 14:57:50.183441 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:50.183448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:50.183574 (+ 126us) server_negotiation.cc:407] Connection header received
0925 14:57:50.183737 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:50.183741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:50.183814 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:50.183896 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:50.184568 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33004: BlockingRecv error: recv got EOF from 127.0.0.1:33004 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 14:57:50.468245 8529 catalog_manager.cc:1259] Loaded cluster ID: b48c6cd297244956ab589e85cd8e27cc
I20260925 14:57:50.468425 8529 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570: loading cluster ID for follower catalog manager: success
I20260925 14:57:50.471254 8529 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570: acquiring CA information for follower catalog manager: success
I20260925 14:57:50.473153 8529 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96b5712655ae4cd5a36bbf36aa687570: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:57:50.713587 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.706774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.17.133.253:45795)
0925 14:57:50.707490 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:50.707498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:50.707790 (+ 292us) server_negotiation.cc:407] Connection header received
0925 14:57:50.707991 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:50.707995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:50.708134 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:50.708257 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:50.708818 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.709693 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.711214 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.711701 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.711851 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:50.712347 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:50.712960 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:50.713136 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:50.713196 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:57:50.713272 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":363,"threads_started":1}
I20260925 14:57:50.725319 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.719326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.17.133.252:39631)
0925 14:57:50.719993 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:50.720000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:50.720019 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:57:50.720121 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:50.720125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:50.720269 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:50.720449 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:50.720869 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.721548 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.722844 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.723256 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.723381 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:50.723952 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:50.724553 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:50.724748 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:50.724891 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 14:57:50.724984 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":405,"threads_started":1}
I20260925 14:57:50.744699 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.731063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.17.133.253:45795)
0925 14:57:50.731255 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:50.731261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:50.737718 (+ 6457us) server_negotiation.cc:407] Connection header received
0925 14:57:50.737904 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:50.737909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:50.738008 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:50.738096 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:50.738765 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.739861 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.742363 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.743073 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.743172 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:50.743802 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:50.743915 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:50.743981 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:50.744228 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 14:57:50.744332 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:57:50.744964 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.731243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.17.133.254:43885)
0925 14:57:50.736954 (+ 5711us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:50.736959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:50.736974 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:57:50.737070 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:50.737073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:50.737241 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:50.737445 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:50.738066 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.739227 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.741472 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.742127 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.742794 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:50.743021 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:50.743501 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:50.744485 (+ 984us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:50.744535 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:57:50.744682 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5611,"thread_start_us":5514,"threads_started":1}
I20260925 14:57:50.749516 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 8250
I20260925 14:57:50.750414 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.738501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.17.133.252:39631)
0925 14:57:50.738668 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:50.738673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:50.744294 (+ 5621us) server_negotiation.cc:407] Connection header received
0925 14:57:50.744436 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:50.744442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:50.744549 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:50.744668 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:50.745226 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.746243 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.748279 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.748915 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.749033 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:50.749607 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:50.749734 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:50.749802 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:50.750048 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 14:57:50.750128 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:57:50.750593 8312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:50.866063 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 8250
I20260925 14:57:50.887132 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 8324
I20260925 14:57:50.888376 8386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:50.993287 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 8324
I20260925 14:57:51.013622 17943 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu pid 8398
I20260925 14:57:51.014712 8460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:57:51.119688 17943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtxnNj/build/asan/bin/kudu with pid 8398
2026-09-25T14:57:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4848 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9209 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (352857 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST